diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b85451e..c5b54f7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -64,7 +64,7 @@ jobs: # Real Proton credentials and /dev/fuse deliberately do not run on pull # requests. Follow docs/TESTING.md on a dedicated test-account runner. - # The seeded simulation runs (docs/MILESTONE-3.0.0.md §8.2): real FUSE + # The seeded simulation runs (docs/TESTING.md): real FUSE # daemons against an in-memory Drive, so they need /dev/fuse but no account. simulation: if: github.event_name != 'pull_request' diff --git a/README.md b/README.md index 61df305..f06c8d0 100644 --- a/README.md +++ b/README.md @@ -149,7 +149,7 @@ keys before it leaves the machine, through a Rust implementation of Proton's Dri Files you open are decrypted into a local cache so programs can read them. What that means for the security of the machine is spelled out in the -[threat model](docs/ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext). +[threat model](docs/ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext). ## Contributing diff --git a/SECURITY.md b/SECURITY.md index b763cd7..3106ba4 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -31,7 +31,7 @@ the interesting boundaries are: or the wrong key material to Proton. What the client deliberately keeps in plaintext on disk, and why, is documented in the -[threat model](docs/ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext). +[threat model](docs/ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext). Behaviour described there is known and not a vulnerability by itself; a way to make it worse than described is. diff --git a/crates/pdfs-core/src/config.rs b/crates/pdfs-core/src/config.rs index 42d8a55..b2d845c 100644 --- a/crates/pdfs-core/src/config.rs +++ b/crates/pdfs-core/src/config.rs @@ -157,7 +157,7 @@ pub struct AppConfig { /// Record a new folder, a new file, a rename or a delete through the mount /// locally and send it from the queue, instead of waiting for Drive in the /// syscall. `None` means on. `false` brings back the old path for one - /// release (docs/MILESTONE-3.0.0.md §10). Defaulted for configs predating + /// release (docs/ROADMAP.md). Defaulted for configs predating /// the field. #[serde(default)] pub local_first: Option, diff --git a/crates/pdfs-core/src/db/migrations.rs b/crates/pdfs-core/src/db/migrations.rs index 208f63b..9eb5cfd 100644 --- a/crates/pdfs-core/src/db/migrations.rs +++ b/crates/pdfs-core/src/db/migrations.rs @@ -1168,8 +1168,8 @@ const MIGRATION_V35: &str = " ALTER TABLE trash ADD COLUMN trashed_at INTEGER; "; -/// Schema v36: every node row has a local id (`lid`), phase 2 of -/// docs/MILESTONE-3.0.0.md. +/// Schema v36: every node row has a local id (`lid`); see +/// docs/ARCHITECTURE.md §2. /// /// The `uid` was the row's key, so a node had no identity until Drive minted /// one, and a create that landed moved its row to a new key. The `lid` is given diff --git a/crates/pdfs-fuse/src/drive.rs b/crates/pdfs-fuse/src/drive.rs index 0816347..1bb7a68 100644 --- a/crates/pdfs-fuse/src/drive.rs +++ b/crates/pdfs-fuse/src/drive.rs @@ -3,7 +3,7 @@ //! `Core` used to hold a concrete [`ProtonDriveClient`], so no test could make //! Drive answer late, answer stale, or not answer at all, and the interleavings //! behind most recent bugs could only be found on a real account over a slow -//! link (`docs/MILESTONE-3.0.0.md` §2, §8.1). [`DriveApi`] is the part of the +//! link. [`DriveApi`] is the part of the //! client the mount, the drain, the event feed and the mirror engine use; the //! real client implements it by forwarding, and the simulation tests implement //! it with an in-memory Drive that injects Drive's known faults. diff --git a/crates/pdfs-fuse/src/sim/fake_drive.rs b/crates/pdfs-fuse/src/sim/fake_drive.rs index eea4812..df05d8c 100644 --- a/crates/pdfs-fuse/src/sim/fake_drive.rs +++ b/crates/pdfs-fuse/src/sim/fake_drive.rs @@ -1,4 +1,4 @@ -//! An in-memory Drive for the simulation tests (`docs/MILESTONE-3.0.0.md` §8.1). +//! An in-memory Drive for the simulation tests (`docs/TESTING.md`). //! //! [`FakeDrive`] is the server: one tree of nodes with uids, revisions and //! names, and an event log. Each simulated daemon reaches it through its own diff --git a/crates/pdfs-fuse/src/sim/mod.rs b/crates/pdfs-fuse/src/sim/mod.rs index b898582..4e4dd3f 100644 --- a/crates/pdfs-fuse/src/sim/mod.rs +++ b/crates/pdfs-fuse/src/sim/mod.rs @@ -1,4 +1,4 @@ -//! The simulation tests of `docs/MILESTONE-3.0.0.md` §8: a fake Drive with +//! The simulation tests of `docs/TESTING.md`: a fake Drive with //! the faults the real one has, and the tools to drive daemons against it. pub(crate) mod daemon; diff --git a/crates/pdfs-fuse/src/sim/model.rs b/crates/pdfs-fuse/src/sim/model.rs index ea96aae..b1b5d9b 100644 --- a/crates/pdfs-fuse/src/sim/model.rs +++ b/crates/pdfs-fuse/src/sim/model.rs @@ -1,5 +1,5 @@ -//! The reference file system of the simulation (`docs/MILESTONE-3.0.0.md` -//! §8.2, property 4): what each syscall a simulated client makes should +//! The reference file system of the simulation (`docs/TESTING.md`, +//! "Simulation runs", property 1): what each syscall a simulated client makes should //! answer, and the tree it should leave, on a POSIX file system. //! //! A [`Model`] covers one client's own folder; paths are relative to it. The diff --git a/crates/pdfs-fuse/src/sim/run.rs b/crates/pdfs-fuse/src/sim/run.rs index b0d1150..a63920d 100644 --- a/crates/pdfs-fuse/src/sim/run.rs +++ b/crates/pdfs-fuse/src/sim/run.rs @@ -1,4 +1,4 @@ -//! The simulation runs of `docs/MILESTONE-3.0.0.md` §8.2. +//! The simulation runs described in `docs/TESTING.md`. //! //! A run starts one or more daemons on a shared [`FakeDrive`], each owning a //! folder `c` that only it writes, so there is one writer per file. A seed diff --git a/crates/pdfs-gui/src/app/pages/browser.rs b/crates/pdfs-gui/src/app/pages/browser.rs index aff3264..d9c4e41 100644 --- a/crates/pdfs-gui/src/app/pages/browser.rs +++ b/crates/pdfs-gui/src/app/pages/browser.rs @@ -832,8 +832,8 @@ fn show_location_column(ui: &Rc, visible: bool) { } } -/// The menu for one entry, in the order of `docs/UI_UX_PLAN.md` §6: open, -/// offline, sharing, organising, and the destructive item last on its own. +/// The menu for one entry, in a fixed order: open, offline, sharing, +/// organising, and the destructive item last on its own. pub(crate) fn entry_context_menu(ui: &Rc, entry: &DirEntry) -> ActionMenu { let mut menu = ActionMenu::new(); let (ui_c, entry_c) = (ui.clone(), entry.clone()); diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 8865f0f..e71a462 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,582 +1,635 @@ # Architecture -This document describes how the Proton Drive Linux client (`pdfs`) is built: the crates, the -filesystem and its caches, the write queue, the sync engine, the control protocol, the thread -layout, what the client writes to disk in plaintext, and the CAPTCHA flow. It is written for -contributors and for anyone auditing the client. User-facing documentation starts at +How the Proton Drive Linux client is built: the crates, the local tree behind the mount, the read +and write paths, the queue that sends changes to Drive, the sync engine for synced folders, the +control protocol, the threads, and what the client writes to disk in plaintext. It is written for +contributors and for anyone auditing the client. User documentation starts at [docs/README.md](README.md). -Numbers quoted here (thread counts, timeouts, limits) are the constants in the source at the time -of writing; the source is authoritative when they disagree. +Numbers quoted here (thread counts, timeouts, limits) are constants in the source. When they +disagree, the source is right. + +- [1. Crates and processes](#1-crates-and-processes) +- [2. The local tree](#2-the-local-tree) +- [3. Read path](#3-read-path) +- [4. Write path](#4-write-path) +- [5. The queue and the drain](#5-the-queue-and-the-drain) +- [6. Remote changes](#6-remote-changes) +- [7. Synced folders](#7-synced-folders) +- [8. Control protocol](#8-control-protocol) +- [9. Threads](#9-threads) +- [10. Threat model: what this client writes to disk in plaintext](#10-threat-model-what-this-client-writes-to-disk-in-plaintext) +- [11. Sign-in and CAPTCHA](#11-sign-in-and-captcha) +- [12. Feature notes](#12-feature-notes) +- [13. Where it is going](#13-where-it-is-going) --- -## 1. Subsystem Overview & Crate Topology - -The application is modularized into four workspace crates, dividing core library logic, filesystem mounting, control-socket IPC, and front-ends. +## 1. Crates and processes ```mermaid graph TD - %% Crates - CLI["crates/pdfs-cli (CLI & Daemon Entrypoint)"] - GUI["crates/pdfs-gui (GTK Front-end)"] - FUSE["crates/pdfs-fuse (FUSE VFS & Sync Loop)"] - CORE["crates/pdfs-core (DB, Cache & IPC Protocol)"] - SDK["proton-sdk-rs (Proton Drive API & Cryptography)"] + CLI["pdfs-cli: pdfs, pdfs daemon"] + GUI["pdfs-gui: pdfs-app, pdfs-tray, pdfs-prompt"] + FUSE["pdfs-fuse: the daemon"] + CORE["pdfs-core: database, cache, config, control protocol"] + SDK["proton-drive-rs: Drive API and cryptography"] - %% Dependencies CLI --> FUSE + CLI --> CORE GUI --> CORE FUSE --> CORE + FUSE --> SDK CORE --> SDK - CLI -.->|Unix Socket IPC| FUSE - GUI -.->|Unix Socket IPC| FUSE + CLI -.->|control socket| FUSE + GUI -.->|control socket| FUSE ``` -### Crate Division & Responsibility Matrix +| Crate | Holds | +|---|---| +| [`pdfs-core`](../crates/pdfs-core) | Authentication and keyring (`auth`), configuration and directories (`config`), the SQLite schema and queries (`db/`), the content cache (`cache`), the control protocol (`control`), search scoring, ignore rules, Google Takeout parsing, the machine profile | +| [`pdfs-fuse`](../crates/pdfs-fuse) | The daemon: FUSE handlers (`filesystem`), the tree (`state`), reads (`reads`), the queue and the drain (`queue`, `drain`, `upload`), the link state (`link`), remote events (`background`), the sync engine (`sync`, `sync/`), moves between locations (`relocate`), control handlers (`control`), push events (`events`), photos, sharing, devices, the supervisor, and the simulation tests (`sim/`) | +| [`pdfs-cli`](../crates/pdfs-cli) | The `pdfs` binary. `pdfs daemon` runs the daemon in-process; every other command is a control-socket client | +| [`pdfs-gui`](../crates/pdfs-gui) | `pdfs-app`, `pdfs-tray` and `pdfs-prompt`. Pages in `src/app/pages/`, widgets in `src/app/widgets/`, CSS and icons in `resources/` | -| Crate | Primary Role | Key Components | State Management | -|---|---|---|---| -| [`pdfs-core`](../crates/pdfs-core) | Core Infrastructure & Services | Cache bookkeeping, database migrations/schemas, IPC protocol payloads, and shared search relevance scoring. | Holds the unified SQLite DB (`Db`) connection and the on-disk cache metadata (`ContentCache`). | -| [`pdfs-fuse`](../crates/pdfs-fuse) | VFS Layer & Reconciliation | FUSE callbacks, background upload queue (`drain`), two-way sync runner. | Manages in-memory inode maps (`State`), active descriptors (`WriteHandle`), and background task threads. | -| [`pdfs-cli`](../crates/pdfs-cli) | Command Line Interface | Command routing, daemon launcher, IPC client wrapper. | Stateless; communicates with daemon over IPC control socket. | -| [`pdfs-gui`](../crates/pdfs-gui) | Graphical Interface | GTK pages, tray, and the resident quick-search prompt. | Keeps UI state only; all durable state and Drive access remain behind the IPC socket. | - -The workspace ships four binaries: `pdfs` (the CLI, and the daemon as `pdfs daemon`) from -`pdfs-cli`, and `pdfs-app`, `pdfs-tray` and `pdfs-prompt` from `pdfs-gui`. The daemon runs as the -systemd user unit `proton-drive.service`. Only the daemon holds the database, the caches and the -API client; every other binary is a client of its control socket. +The daemon runs as the systemd user unit `proton-drive.service`. Only the daemon holds the +database, the caches and the API client. Every other program is a client of its control socket +(§8) and never opens the database or calls Drive itself. --- -## 2. In-Memory VFS State & File Operations +## 2. The local tree -The VFS layer implements FUSE via the `fuser` crate. Because the remote storage contains base64-encoded file keys and requires cryptographic envelope parsing, raw listings and inodes are virtualized and stored in a local state directory. +### Nodes and local ids -### Inode and Path Resolution -* **In-Memory Cache (`State`):** Maps FUSE `u64` inodes to Proton Drive `NodeUid`s. -* **Database Row Mapping (`StoredNode`):** Stores directories, sizes, and timestamps. -* **On-Demand Loading (`ensure_children`):** If a directory is accessed, the daemon checks its database `listed` flag. If `listed = 0`, it triggers an API call to fetch remote nodes, populates the DB and in-memory caches, and returns. +Every node the daemon knows is a row in the `nodes` table of `cache.db`. Each row has a **local +id** (`lid`), given when the row is made and never changed. Rows link to their parent by +`parent_lid`, so a folder's children, its subtree and its ancestors are found by local id. -```mermaid -sequenceDiagram - autonumber - actor User as Kernel (VFS Call) - participant FUSE as pdfs-fuse VFS - participant ST as State (In-Memory) - participant DB as Db (SQLite) - participant API as Proton API Client +- **Inode = local id.** A node's inode is `lid + 1` (`lid_ino` in `state.rs`). It is the same in + every mount and after a restart, so the kernel's cached entries never point at an inode the + daemon has forgotten. +- **Nodes made here.** A node created on this machine has no Drive uid until its create lands. It + goes by the stand-in `local~` (`db::local_uid`), a function of its row. When the create + lands, the row gets the real uid in the same transaction. +- **One tree for every mount.** The daemon holds one in-memory `State`, a cache of the `nodes` + table. `~/ProtonDrive` and each online-only synced folder are FUSE sessions rooted at different + nodes of that tree; each session calls its own root inode 1 and translates at the kernel + boundary. - User->>FUSE: lookup(parent_ino, "report.pdf") - FUSE->>ST: children.get(&parent_ino) - alt Parent listing is resident in-memory - ST-->>FUSE: returns child_ino - else Listing missing in-memory - FUSE->>DB: children_if_listed(parent_uid) - alt Parent marked listed in DB - DB-->>FUSE: returns child node metadata list - FUSE->>ST: intern_from_db() and populate children cache - else Parent not listed in DB - FUSE->>API: enumerate_folder_children_node_uids() - API-->>FUSE: list of UIDs - FUSE->>API: enumerate_nodes(uids) - API-->>FUSE: list of decrypted Nodes - FUSE->>DB: upsert_nodes() & set_listed(true) - FUSE->>ST: intern_batch() and populate children cache - end - end - FUSE-->>User: returns child inode metadata (attributes, TTL) -``` +### Listings on demand -### Access Classification and Enforcement +A folder is listed from Drive the first time it is opened. `ensure_children` checks the folder's +`listed` flag. If it is set, the children come from the database. If not, the daemon enumerates +the folder from Drive, writes the children to the database, sets the flag, and interns them. -Every `Entry` carries an `Access` (`Owner | Editor | Viewer | Unknown`), inherited from its parent at intern time rather than resolved per node — a child is always interned from its parent's listing, so one edge lookup answers it. A known share root takes its access from the persisted `share_access` table instead, which is also what makes the classification correct offline. +### Access -Two rules decide the cases inheritance cannot: +Every entry carries an `Access` (`Owner`, `Editor`, `Viewer` or `Unknown`). It is inherited from +the parent when the entry is interned. A share root takes its access from the persisted +`share_access` table, which also keeps the classification right offline. Two rules cover what +inheritance cannot: -* **Not under a share, no role → `Owner` (fail open).** My Files and device folders are owned content; regressing this denies ordinary writes. -* **Under a share with no usable role → `Viewer` (fail closed).** An unrecognised permission mask is never degraded into a guess. +- **Not under a share, no role: `Owner`.** My files and device folders are the user's own. +- **Under a share, no usable role: `Viewer`.** An unknown permission mask is never guessed upward. -A node whose parent is not resident is the awkward case: a device folder's parent is the device root, which is never persisted as a node, so the whole subtree hydrates parentless. It resolves the way the persisted authority (`Db::effective_node_access`) does — a recorded share row above it wins, otherwise fail open on the mount's own volume and closed on a foreign one (`docs/BUGS.md` B79). +A node whose parent is not resident resolves as `Db::effective_node_access` does: a recorded share +above it wins; otherwise it is writable on the mount's own volume and read-only on a foreign one +(`docs/BUGS.md` B79). -Enforcement is three layers, and only the third closes B34: +Writes are refused in three layers: -1. **Mode bits.** `attr()` returns `0o555`/`0o444` for a non-writable entry, and both mount paths set `MountOption::DefaultPermissions`, so the *kernel* refuses `open(O_WRONLY)`, `access(W_OK)` and namespace operations for any non-root process. -2. **Handler gates.** `EACCES` in `create`/`mkdir`/`unlink`/`rmdir`/`rename` (parent-writable; both parents for a rename) and in `open`-for-write/`write`/`setattr`/`fallocate`. Covers root and stale attribute TTLs. -3. **Queue guards.** `Core::require_uid_writable` admits a mutation only when the persisted authority *and* every live inode space agree the uid is writable. Nothing reaches `pending_op` otherwise, so the perpetual failing drain that B34 describes cannot occur even if a handler check is missed. The intersection is across mounts because a uid may be resident in more than one inode space. +1. **Mode bits.** A non-writable entry reports `0o555` or `0o444`, and every mount sets + `DefaultPermissions`, so the kernel refuses writes from any non-root process. +2. **Handler checks.** `create`, `mkdir`, `unlink`, `rmdir`, `rename` (both parents), and opens + for writing, `write`, `setattr` and `fallocate` answer `EACCES`. This covers root and stale + attribute caches. +3. **Queue guard.** `Core::require_uid_writable` admits a change only when the database and the + live tree both agree the node is writable. Nothing reaches the queue otherwise, so a refused + write cannot become an op that fails forever (`docs/BUGS.md` B34). -`EACCES`, not `EROFS`, for a read-only subtree inside a read-write mount: `EROFS` means "read-only filesystem" and misleads the heuristics in `cp`, `rsync` and `git`. +`EACCES`, not `EROFS`: a read-only subtree inside a read-write mount is not a read-only +filesystem, and `EROFS` misleads `cp`, `rsync` and `git`. -### Local Locations +### Locations -`mount` is a presentation table: one row per local place this client occupies — the primary My Files session, and each device folder in mirror or on-demand mode. `sync_folder` remains the sync engine's own table (`sync_entry` is FK'd to it) and `MountKind::Device { sync_folder_id }` is the join. `Request::ListLocations` serves the table to `pdfs locations` and the GUI's Locations page. - -A mirror folder is a plain local directory with no FUSE session, so a row can legitimately describe a location that is not mounted — which is why the page is called *Locations* rather than *Mounts*. The primary mountpoint stays in `AppConfig`, with its `mount` row written at daemon start as a projection of it, so there are not two sources of truth for that path. +The `mount` table has one row per local place the daemon occupies: My files and each synced +folder, in either mode. It serves `pdfs locations` and the app. A mirrored folder is an ordinary +directory with no FUSE session, which is why the table speaks of locations, not mounts. The My +files mountpoint stays in `config.json`; its row is written at start as a copy of it. --- -## 3. Read Path & Block Caching Pipeline +## 3. Read path -Reads are served from the revision's content blocks. `Core::read_range` looks in three places +`Core::read_range` serves a read from the revision's content blocks. It looks in three places before the network: -1. **Whole-file blob.** A file kept available offline is downloaded completely into - `ContentCache` and read from there. -2. **Stream ring.** For every other file, a bounded in-memory ring (`RING_BYTES`, 128 MiB) of - recently decrypted blocks is checked first, because the kernel reads in pieces far smaller - than a block. -3. **Block cache.** Blocks are then looked up on disk, under `content/blocks/`. They count - against `cache_budget` and are evicted least-recently-used. +1. **Whole-file blob.** A file kept available offline is downloaded completely into the content + cache and read from there. +2. **Stream ring.** An in-memory ring of recently decrypted blocks (`RING_BYTES`, 128 MiB). The + kernel reads in pieces far smaller than a block. +3. **Block cache.** Blocks on disk under `content/blocks/`. They count against `cache_budget` and + are evicted least recently used first. On a miss, only the blocks that overlap the request are fetched, through a `RevisionReader` kept -open per revision (`MAX_OPEN_READERS = 64`, validated by `(mtime, size)`). +open per revision (`MAX_OPEN_READERS = 64`). A fetch that cannot finish within +`READ_FETCH_TIMEOUT` (120 s) fails the read with `EIO`. -* **Block geometry.** Block boundaries come from the revision itself. The first read of a file - assumes uniform `BLOCK_SIZE` (4 MiB) blocks; once a reader is open, the real block sizes are - recorded (`store_block_geometry`) and later reads plan on them. Proton does not guarantee - 4 MiB blocks, and assuming it served wrong bytes (`docs/BUGS.md` B85). -* **Read-ahead.** A read that proves sequential prefetches a window of 2 to 8 blocks - (`PREFETCH_MIN`, `PREFETCH_MAX`). Prefetch takes a permit from a global budget of 8 and gives up - instead of queueing, so it never delays a demand read. -* **Large videos.** An unpinned video of 256 MiB or more (`STREAM_BYPASS_MIN`) streams without - persisting its blocks, so watching a film does not evict the rest of the cache. +- **Block geometry.** The first read of a file plans on uniform 4 MiB blocks (`BLOCK_SIZE`). Once + a reader is open, the real block sizes are stored (`store_block_geometry`) and later reads plan + on them. Proton does not guarantee 4 MiB blocks (`docs/BUGS.md` B85, B87). +- **Read-ahead.** A sequential reader gets 2 to 8 blocks of prefetch (`PREFETCH_MIN`, + `PREFETCH_MAX`). Prefetch takes a permit from a budget of 8 (`PREFETCH_BUDGET`) and gives up + instead of waiting, so it never delays a read someone asked for. +- **Large videos.** An unpinned video of 256 MiB or more (`STREAM_BYPASS_MIN`) streams without + storing its blocks, so watching a film does not evict the rest of the cache. -```mermaid -sequenceDiagram - autonumber - actor Kernel as Kernel Read (offset, size) - participant FUSE as pdfs-fuse VFS - participant Cache as ContentCache (Local Disk) - participant Ring as StreamRing (In-Memory) - participant API as Proton API Client - - Kernel->>FUSE: read(ino, fh, offset, size) - alt Whole-file blob cached (available offline) - FUSE->>Cache: read blob range - Cache-->>FUSE: bytes - else - FUSE->>FUSE: block_geometry() spans overlapping [offset, offset+size) - loop For each span - alt Span in stream ring - FUSE->>Ring: get(span) - Ring-->>FUSE: block bytes - else Span in block cache - FUSE->>Cache: read_block(span) - Cache-->>FUSE: block bytes - else Miss - FUSE->>API: RevisionReader.read_at(span) - API-->>FUSE: decrypted block bytes - opt Not a large unpinned video - FUSE->>Cache: store_block(span) - end - FUSE->>Ring: insert(span) - end - end - end - FUSE->>FUSE: stitch blocks and slice to offset/size - FUSE-->>Kernel: return data buffer -``` +A file open for writing is read from its handle: written bytes from the scratch file, the rest +from its base. --- -## 4. Write Path & Staging/Draining Pipeline +## 4. Write path -Because Proton Drive does not support partial byte writes, modified files must be uploaded as whole new revisions. +Since 3.0.0 the mount is **local-first**. A syscall changes the local tree and the queue, and +returns. It does not wait for Drive, except to read content that is not cached. Being offline +only means more work is queued. -1. **Staging writes (`WriteHandle`):** Writes are stored locally in a `scratch` file. The daemon tracks modified regions using `Intervals` (which holds ranges of edited bytes). -2. **Close/Release (`queue_revision`):** When the application closes the file descriptor, the daemon: - - Fetches any untouched gaps from the remote base file to compile the full file. - - Durably publishes the scratch data and authored-range sidecar into `staging` under a `{uid}-{millis}-{counter}` name. Temporary data and metadata are synced before atomic rename, and their directory is synced before the source is removed. - - Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission. The debounce is adaptive (`Core::revision_debounce`): it starts at `DRAIN_REVISION_DEBOUNCE = 2s` and, once a node has been uploaded, widens toward how long that upload actually took, bounded by `DRAIN_REVISION_DEBOUNCE_MAX = 60s`. A file saved faster than it can be sent therefore supersedes in the queue rather than mid-upload. - - Signals `Core::cancel_upload` for the node before touching the queue. A drain worker may already be reading the blob this write supersedes; the flag is read by the upload's `CountingReader`, which refuses the SDK's next block. `queue_trash` and `discard_queued_ops` do the same, because they unlink the blob outright. -3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 16` background workers share the operations queue; one that lands an op wakes the idle ones, since it may have made others claimable. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes. +### Namespace changes -```mermaid -sequenceDiagram - autonumber - actor Kernel as Kernel Write (fh, offset, data) - participant FUSE as pdfs-fuse VFS - participant WH as WriteHandle (Scratch File) - participant DB as Db (SQLite Queue) - participant DR as Drain Thread - participant API as Proton API Client +`mkdir`, `create`, `rename`, `unlink` and `rmdir` through the mount: - Kernel->>FUSE: write(ino, fh, offset, data) - FUSE->>WH: write_at(offset, data) - FUSE->>WH: update written intervals - FUSE-->>Kernel: return bytes_written - - Note over Kernel, FUSE: Application closes file (close(2)) - Kernel->>FUSE: release(fh) - FUSE->>FUSE: fill_gaps() (fetch untouched remote ranges) - FUSE->>FUSE: move scratch file to staging directory - FUSE->>DB: enqueue_op(OP_REVISION, staged_path, meta) - FUSE->>FUSE: record_pending_write() (update size/mtime in memory & DB) - FUSE-->>Kernel: return success (async release) - - Note over DB, DR: Background Queue Processing - DR->>DB: next_due_op() - DB-->>DR: return OP_REVISION - DR->>API: upload_new_revision_from(staged_path) - API-->>DR: return new node revision metadata - DR->>DB: delete_op() - DR->>FUSE: refresh_after_upload() (sync local metadata with server time) -``` - ---- - -## 5. Sync Engine (Two-Way Reconciliation) - -The sync engine handles offline-capable, bidirectional synchronization between the local disk and Proton Drive for directories marked in `mirror` mode. - -### Lifecycle of a Sync Pass -1. **Walk Local:** Walks the local directory tree recursively, scanning sizes and modification times while carrying a completeness result. A `readdir`, metadata, permission, or transient I/O failure makes the pass non-destructive instead of turning omitted paths into deletions. -2. **Walk Remote:** Walks the remote database representation. If remote file modification times are updated, it calls the API to decrypt their sizes. -3. **Load Baseline:** Loads the `sync_entry` database table, which contains the snapshot of both sides during the *last successful sync*. -4. **Permutation Diffing:** The loop compares the three states (`local`, `remote`, `baseline`) to classify items: - -```mermaid -graph TD - %% States - Classify{"Classify (Local, Remote, Baseline)"} - - %% Logic Rules - Classify -->|Both Sides Match| Match["No-Op (In Sync)"] - Classify -->|Local Changed, Remote Untouched| Upload["Upload Revision"] - Classify -->|Remote Changed, Local Untouched| Download["Download Revision"] - Classify -->|Both Sides Changed| Conflict["Conflict Copy (Local renamed to 'sync-conflict', remote downloaded)"] - Classify -->|Local Deleted, Remote Untouched| RemoteDelete["Trash Remote Node"] - Classify -->|Remote Deleted, Local Untouched| LocalDelete["Delete Local File"] - Classify -->|New Local File, No Remote/Baseline| UploadNew["Upload New Node"] - Classify -->|New Remote File, No Local/Baseline| DownloadNew["Download New Node"] -``` - -5. **Depth-Ascending Batching:** Folders are processed first to ensure hierarchies exist before files are placed. Work is executed concurrently up to a set limit. -6. **Safety Gate and Settle:** Destructive plans are rejected when the scan is incomplete. Total-wipe protection applies to every non-empty baseline, including a single-entry folder. On success, baseline entries are upserted, timestamps updated, and any pending mode switches (e.g. going on-demand) are evaluated. Failed local deletion or conflict preservation retains the previous baseline and prevents remote content from overwriting the local source. - -### Conflict Copies - -A conflict keeps both versions. The local file is renamed to -`{stem} (sync-conflict {unix_secs}).{ext}` (with `-1`, `-2`, … appended to the timestamp when the -name is taken) and the remote version is downloaded under the original name. The conflict sweep -(§7) later trashes copies proven byte-identical to their sibling when `conflict_sweep` is -`enforce`. - -### Devices and the Machine Profile - -Synced folders live under a *device*: a per-computer root in Proton Drive. `Core::ensure_device` -decides which device this machine is: the adopted `device_uid` from `config.json` when set, -otherwise a Linux device whose name equals the hostname, otherwise a newly registered one. An -adopted UID that no longer exists is reported, never silently replaced. - -Because the local database does not survive a lost machine, the daemon backs up the machine's -arrangement to `/.proton-drive-linux/profile.json` (`pdfs_core::profile`): each synced -folder's remote UID, local path and mode, the pins, and the ignore patterns. It goes up the normal -upload path and so is end-to-end encrypted like any other file. Mutating requests call -`Core::touch_profile`, which coalesces a burst of changes into one upload. `PROFILE_VERSION` -guards the format: a reader refuses a profile from a newer version instead of applying part of it. - -Restore (`ListRestorableFolders`, `RestoreSyncFolders`, and the `Device` variants for another -computer) lists the device root's folders, proposes a local path from the profile when its parent -exists on this machine and `~/` otherwise, and on confirmation adds a `sync_folder` row per -folder bound to the remote UID. The download is an ordinary reconcile against an empty baseline; -on-demand mode is applied through the normal mode-switch request. The user guide is -[RECOVERY.md](RECOVERY.md). - ---- - -## 6. IPC Socket Protocol - -The CLI and GUI front-ends do not access database files or make network calls directly. They communicate with the background daemon process over a Unix domain socket. - -* **Transport:** IPC over Unix Stream Socket. -* **Framing:** Line-delimited JSON payloads. -* **Resource limits:** Requests are limited to 1 MiB, must end with a newline, and must arrive within 10 seconds. At most 64 connection handlers are active at once. -* **Control Protocol:** - * Client sends a single JSON line (`Request`). - * Daemon parses, handles the request, and replies with a single JSON line (`Response`). - * Timeout durations are separated: **2 seconds** for writes (avoids hangs on defunct sockets) and **120 seconds** for reads (accommodates heavy transfers). Status polls use a 5-second read timeout, which is what the app reports as "Not responding". -* **Errors:** A failed request replies with `Response::Error { message, kind }`. The `kind` is machine-readable, and `pdfs --json` passes it through. -* **Compatibility:** After an upgrade, the old daemon keeps running until the service restarts, so front ends must cope with a daemon that does not know a newer request. It answers such a request with an error instead of dropping the connection. New fields are added with `#[serde(default)]`, and new variants get a separate request rather than an optional field an old daemon would ignore. - -### Push Events - -Since 2.5.0 the front ends do not poll. The app and the tray each send `Request::Subscribe` once -and keep the connection open; the daemon then streams one `Event` per line: - -| Event | Sent when | Payload | +| Call | What happens locally | Queued op | |---|---|---| -| `Status` | On subscribe, and whenever any status field changes | A full `Response::Status` | -| `Transfers` | On subscribe, and whenever the transfer snapshot changes | Transfers and background jobs | -| `Changed { topics }` | Something a page shows changed | `Topic`s to re-read: `files`, `photos`, `trash`, `shares`, `devices`, `locations`, `queue`, `conflicts`, `activity` | -| `Heartbeat` | Every 10 seconds of silence | — | +| `mkdir`, `create` | A new row under `local~` (`queue_local_node`) | `mkdir` or `create` | +| `rename` | The node moves in the tree at once (`queue_rename`) | `rename`, holding the end state and the node's original parent and name | +| `unlink`, `rmdir` | The node leaves the tree; ops queued for it are dropped (`queue_trash`) | `trash`, or nothing for a node Drive never saw | -Code that changes state publishes its topics to the `EventHub` (`crates/pdfs-fuse/src/events.rs`); -the control handler maps each mutating request to the topics it touches. Changes within 150 ms -of each other are coalesced into one event. A new subscription is told every topic changed, so it -catches up on anything it missed. At most 8 subscriptions are served at once. +A rename or move asked for over the control socket (`pdfs rename`, `pdfs move`, the app) is queued +the same way, and always when the node already has something queued, so it cannot overtake it +(`docs/BUGS.md` B149). -On the client side, `pdfs_core::control::follow` reconnects across daemon restarts with backoff -and treats 25 seconds without a line as a stuck daemon. A daemon older than 2.5.0 answers -`Subscribe` with an error; `follow` then falls back to polling status and transfers, so the front -end sees the same events, only later. +`"local_first": false` in `config.json` brings back the old path for this release: while online, +these calls go to Drive inside the syscall. It is removed in the next release (§13). -### Unified Search +### Content -`SearchV2` is the shared search boundary for the resident prompt. One request carries the query, result limit, requested sources (`Drive`, `Local`), and content kind. The daemon queries Drive metadata and the local home-directory index, then applies [`pdfs_core::search::relevance_score`](../crates/pdfs-core/src/search.rs) to both result sets. Exact and prefix matches rank above substring, abbreviation, and bounded typo matches; all query terms must match either the basename or parent path. Returning scores on one scale lets the GUI merge both sources into a deterministic **Best matches** list. +1. **Open for writing.** A `WriteHandle` with a scratch file under `content/scratch/`. An interval + set records which bytes were written; reads of the rest go to the base. +2. **Release** (`close(2)` of the last handle). Gaps the program did not write are filled from the + base, from the cache when it has them. The scratch file and its metadata are published into + `content/staging/` (fsync, atomic rename, directory fsync), and a `revision` op is queued in + the same transaction that supersedes an older one. A new file's bytes ride on its `create` op + instead. +3. **Debounce.** A `revision` op waits `DRAIN_REVISION_DEBOUNCE` (2 s) before it is sent, so a + quick follow-up write replaces it in the queue. Once a file has been uploaded, the wait widens + toward how long that upload took, up to `DRAIN_REVISION_DEBOUNCE_MAX` (60 s) + (`Core::revision_debounce`). +4. **Cancel.** Before the queue is touched, `Core::cancel_upload` stops an upload already reading + the blob that is being replaced. -The prompt is a single-instance GTK application. It retains and hides its window between activations, resets its query state when summoned, and ignores stale asynchronous replies. Folder and streamable audio/video results open through the FUSE mount so applications can issue range reads; ordinary Drive files use the daemon's materialize-then-open path. +`fsync` means "durable locally". Writes that were fsynced but not yet released when the daemon +died are found at the next start (`recover_fsynced_writes`), kept in `content/recovery/`, and +queued. + +### Transient names + +A file under a scratch name, such as a browser's `.crdownload` or `.part`, or an editor's `.swp`, +is created locally and its op is **parked** (`PARK_UNTIL`). The rename to the finished name +releases it, so only the finished file reaches Drive (`docs/BUGS.md` B70). A park still standing +after an hour (`PARK_EXPIRY_MS`) is released anyway: bytes the user can see are bytes the user +expects on Drive. --- -## 7. Subsystem Interaction & Thread Map +## 5. The queue and the drain -The background daemon relies on the following thread topology: +### Ops -1. **Main Thread / Dispatch Loop:** Blocks on `fuser::Session` loop. Reads kernel FUSE events and hands off network-bound VFS work to the FUSE workers pool. Each on-demand synced folder is a further FUSE session with its own dispatch loop and its own inode space (`State`). -2. **FUSE Workers Pool (11 threads, two lanes):** Bounded thread pool handling network operations. Split into 3 threads reserved for metadata (`lookup`, `readdir`) and 8 general threads that serve transfers (block reads) and fall back to metadata when no transfer is waiting. Reserved threads never accept a transfer — that is what keeps a directory listing from queuing behind saturated downloads (audit A6). -3. **Control Threads (`pdfs-control`):** Listen on the Unix socket and admit at most 64 concurrent, timeout-bound handlers. Long-lived subscriptions hold one handler each, up to 8 (see §6). -4. **Sync Engine Threads (`pdfs-sync`, `pdfs-sync-poll`):** Serialize sync passes. Wake on debounced inotify changes, the periodic poll, remote events, or a user request. -5. **Drain Workers (`pdfs-drain-0` … `pdfs-drain-15`):** `DRAIN_WORKERS = 16` threads share the queue of staged writes and changes (`PendingOp`), with per-node ordering (§4), and retry failures with exponential backoff. Worker 0 also runs the queue's idle chores. -6. **Conflict Sweep Thread (`pdfs-conflict-sweep`, optional):** Reconciles leftover `(sync-conflict …)` copies — 30 s warmup, then one pass every 5 minutes. A copy proven identical to its live sibling (equal size **and** equal `content_sha1`) is redundant; anything it cannot prove identical is surfaced to the activity feed and left alone. Governed by `AppConfig.conflict_sweep` / `PDFS_CONFLICT_SWEEP` (`SweepMode`): **report-only by default**, `off` skips the thread entirely, and only `enforce` lets it trash. Because it deletes, the enforcing path re-verifies revision id, size, digest, name, parent, queued ops and open handles immediately before acting rather than trusting its own pass-start snapshot. See `docs/BUGS.md` B69 and B71. -7. **Remote Event Tasks (Tokio):** `run_event_sync` follows Proton's event stream for the Drive volume and applies each change to every mounted inode space, invalidating kernel entries as it goes. The cursor is persisted after every batch, so changes made while the daemon was stopped are applied on the next start. `run_photos_event_sync` follows the photos volume separately; it only drops gallery rows or the timeline's freshness stamp, since photos are not part of the mount. -8. **Event Sampler (`pdfs-events`):** Publishes status, transfer and sync-progress changes to subscribed front-ends (§6). -9. **Supervisor (`pdfs-supervisor`):** Logs workers that hold one job for minutes, a queue that stops draining and control requests that never return; samples memory; and pings the systemd watchdog only after a real round trip over the control socket, so a hung daemon is restarted (`WatchdogSec=120`). `pdfs diagnostics` reports the same data on demand. -10. **Smaller helpers:** `pdfs-online-probe` (connectivity), `pdfs-localindex` (home-directory index for the search prompt), `pdfs-pause` (ends a timed pause), `pdfs-conflict-sweep`, and `pdfs-similar` (duplicate-photo detection, on request). +The queue is the `pending_op` table. An op is a change accepted locally that Drive has not seen +yet. -Every long-lived loop waits on one shared `Shutdown` signal instead of sleeping, so an in-process remount joins the old generation of threads instead of leaking it (`docs/BUGS.md` B44). - ---- - -## 8. Threat Model: What This Client Writes to Disk in Plaintext - -Proton Drive is zero-knowledge: the server never holds the keys to your content. That property ends at this daemon. Serving a remote file through a POSIX filesystem means producing plaintext, and serving it *quickly* means keeping some of that plaintext around. This section states exactly what lands on disk, because the guarantee users infer from "zero-knowledge" is stronger than the one a files-on-demand client can offer locally. - -**The short version: the cache and state directories hold decrypted content and decrypted metadata, and this client assumes the disk underneath them is encrypted (LUKS, or an encrypted home).** On an unencrypted disk, an attacker with the powered-off machine can read cached file content and the full name/structure of your Drive without ever touching your password. - -### 8.1 Decrypted content - -Everything under `$XDG_CACHE_HOME//content/` is plaintext: - -| Path | Holds | Lifetime | +| Kind | Does | Carries | |---|---|---| -| `` blobs | Whole decrypted files (pinned files, opened files) | Until LRU eviction or budget purge | -| `blocks/` | Decrypted 4 MiB block ranges of partially-read files | Until LRU eviction | -| `thumbs/` | Decrypted thumbnails and previews | Until LRU eviction | -| `scratch/` | In-progress writes from open file handles | Until `release`, or rescued at next open | -| `staging/` | Released writes awaiting upload | **Until the upload lands** | -| `recovery/` | `fsync`ed writes rescued from an unclean shutdown | **Until replayed into `staging/`** | +| `create` | Makes a file | Parent and name; the staged bytes | +| `mkdir` | Makes a folder | Parent and name | +| `revision` | Uploads new content | The staged blob and the revision it was based on | +| `rename` | Moves and renames | The end state (parent, name), and the original parent and name | +| `trash` | Trashes a node | Its name | -`staging/` and `recovery/` deserve separate attention: unlike the cache directories, they are not a copy of something the server already has. They hold user-authored content that may exist **nowhere else yet**, which is why they are deliberately never cleared on startup (§4, and audit A2). They are simultaneously the most sensitive thing on disk and the thing that must not be deleted to reclaim space. +Each op names its node and parent by local id (`lid`, `parent_lid`) as well as by uid. +`revision`, `rename` and `trash` describe an end state, so a newer one replaces an older one for +the same node (`op_supersedes`). `create` and `mkdir` are never replaced. -### 8.2 Decrypted metadata +### Two invariants -`$XDG_STATE_HOME//cache.db` is a plain SQLite database containing **decrypted node names**, the folder hierarchy, sizes, timestamps, a trigram full-text index over those names, the activity log, and the photos timeline. It is not evictable and not budgeted — it is the persistence layer the in-memory tree rehydrates from (§2). +From `drain.rs`: -Filename and directory-structure confidentiality is an explicit part of Proton Drive's model (each folder's manifest is encrypted server-side). This database is where that property is spent locally: it is a queryable, plaintext index of your entire Drive, and it survives cache purges. A `PurgeCache` clears content, not this. +1. **A staged blob is the only copy of the user's bytes.** It is deleted only after its op has + provably landed, never on a path that may be retried. +2. **A failure never stops the queue.** A failed op gets a later `next_attempt_at` (exponential + backoff), so one stuck file cannot hold up the others. -### 8.3 What is *not* written in plaintext +### The drain -Credentials. The session blob — access and refresh tokens, and the key material needed to resume unattended — lives only in the OS keyring via libsecret (`auth.rs`), never on disk in cleartext. `config.json` and `pins.json` hold settings and node uids, no secrets. +`DRAIN_WORKERS = 16` threads share the queue. A worker takes work with +`Db::claim_next_due_op`, which selects and marks an op in one transaction. It skips an op when: -Note that the *control socket* is a credential of a different kind: anything that can connect to `control.sock` can drive the daemon — list and read paths, upload, trash, create share links — without touching the keyring at all. See §8.5. +- it is not due yet (debounce, backoff, or parked); +- another worker holds an op for the same node, so each node's ops land in the order they were + made; +- its parent is still a `local~` stand-in, so a child's create waits for its folder's create. -### 8.4 Memory, swap, and the page cache +A worker that lands an op wakes the idle ones, because the landing may make other ops claimable. +Claims are process-local: the database is single-writer (a `flock`), so a claim found at start +belongs to a crashed run and `Db::clear_op_claims` drops it. -Two exposures this client does **not** currently mitigate, stated plainly rather than left implied: +A landing create records the real uid on the node's row. Ops inside a landed folder need no +rewrite: they find the parent's uid through `parent_lid`. -- **Swap.** Content keys, session keys, and decrypted buffers live in ordinary heap memory. Nothing calls `mlock(2)`, so under memory pressure they may be paged out. Raising `LimitMEMLOCK` in the systemd unit would *not* change this — there is no locking to permit. The effective mitigation is encrypted swap (dm-crypt / `systemd-cryptsetup`), which is standard on a LUKS install. -- **Kernel page cache.** Plaintext returned through FUSE is cached by the kernel like any other file data, and is likewise swappable. Defeating this would mean `direct_io` on every read, forfeiting the readahead and caching that make the mount usable. The trade is taken deliberately in favour of performance. +### Conflicts -### 8.5 Enforced ownership and file modes +A `revision` op stores the revision it was based on. If Drive's current revision differs and is +not one this daemon sealed itself (`own_sealed_rev` table), the queued bytes are kept as a +conflict copy (`Core::revision_conflict`) next to the remote version. A name taken on Drive by a +node this machine has not seen lands under a conflict name too. Both go to the activity log. -On every start, `AppDirs::ensure` verifies that the state, cache, and configuration paths are real directories owned by the effective user, rejects symlink substitution or the wrong owner, and enforces mode `0700`. Both Unix sockets are changed to `0600` immediately after binding; failure is fatal rather than falling back to an unguarded daemon. +Conflict copies are named `{stem} (sync-conflict {unix_secs}).{ext}`. -These checks protect the artifacts even when the surrounding home or XDG parent is traversable: +### Failures and sync issues -| Artifact | Why access is restricted | +An op that has failed `FAILING_ATTEMPTS` (6) times counts as failing. An op that Drive refused +gets a **sync issue** at once, stored on the op: + +| Issue | Meaning | |---|---| -| `content/` | Another local user can read cached plaintext file content | -| `cache.db` | Another local user can read the full decrypted name/structure index | -| `control.sock` | **Another local user can drive the daemon**: enumerate, read, upload, trash, create public share links | +| `quota` | The account's storage is full | +| `access` | The account may no longer change the node, for example a revoked share | +| `missing` | The node or its folder is gone from Drive | +| `limit` | Too many items in the folder, or nested too deep | +| `rejected` | Drive refused the change, for example a name it does not accept | -The socket is the sharpest of the three, because it is an authority boundary rather than a data one — connecting to it confers the daemon's authenticated session without any credential. +The op stays queued and keeps retrying; nothing is deleted. The user sees the issue in +`pdfs sync issues`, the tray, **Sync → Overview**, and as a badge on the node in the file browser. +They can save the content (`pdfs sync export`) or drop the change and undo it locally +(`pdfs sync discard`). Only a discard the user asks for removes an op. -Configuration publication follows the same fail-closed model: a restricted temporary file is written and synced, atomically renamed, and followed by a directory sync. A malformed existing configuration is reported and preserved rather than overwritten with defaults. +### Link state -See [RECOVERY.md](RECOVERY.md) for what a lost machine means for the plaintext described in this section, and what to revoke. - -### 8.6 Implications for deployment - -- Treat the cache and state directories as being as sensitive as the Drive contents themselves. -- Restored or manually copied profiles must remain owned by the user. The daemon refuses to start against wrong-owner or symlinked sensitive directories. -- Purging the cache (`pdfs` settings, or `PurgeCache` over IPC) removes content but **not** `cache.db`, and deliberately never removes undrained `staging/` or `recovery/` blobs. +`link.rs` tracks whether Drive is reachable. A network error marks the daemon offline, and a probe +thread (`pdfs-online-probe`) backs off until a request succeeds. Remote calls a caller waits on +have a deadline (`INTERACTIVE_CALL_TIMEOUT`, 20 s); uploads get one that grows with their size +(`link::upload_deadline`). --- -## 9. Human Verification (CAPTCHA) Flow +## 6. Remote changes -When logging in from an unfamiliar IP address or VPN, the Proton API may gate the sign-in with a human verification challenge (CAPTCHA). This client handles this asynchronously and interactively. +`run_event_sync` follows Proton's event stream for the Drive volume. Each event updates the +database and the tree, and the kernel is told about entries that changed. The cursor is stored as +events are applied, so changes made while the daemon was stopped arrive at the next start. An +event for a revision in `own_sealed_rev` is our own echo, not a change from elsewhere. -### 9.1 Sequence of Verification and Re-Authentication +`run_photos_event_sync` follows the photos volume with its own cursor. Photos are not in the mount, +so it only drops gallery rows or marks the timeline stale (`TIMELINE_TTL`, 60 s). + +`pdfs refresh` drops a cached listing so the next read fetches it again. + +--- + +## 7. Synced folders + +A synced folder in **mirror** mode is a plain local directory kept in step with a folder under +this computer's device in Drive. It has its own engine (`sync.rs`, `sync/`); it does not use the +queue of §5 yet (§13). A folder in **online-only** mode is a FUSE session on the tree (§2), and +its changes go through the queue like My files. + +### A sync pass + +A filesystem watcher (`notify`) and a remote poll every 120 s feed a debounced pass (2 s, at most +30 s). Passes are serialized per daemon. A pass: + +1. **Walks the local tree.** Any `readdir`, metadata or permission failure marks the scan + incomplete, and an incomplete scan cannot delete anything. +2. **Walks the remote tree.** +3. **Loads the baseline**, the `sync_entry` rows: both sides as of the last successful pass. +4. **Classifies each path** by comparing local, remote and baseline, with `(mtime, size)` as the + change signal: + + | Local | Remote | Action | + |---|---|---| + | changed | unchanged | upload | + | unchanged | changed | download | + | changed | changed | conflict: the local file becomes a conflict copy, the remote one is downloaded | + | deleted | unchanged | trash on Drive | + | unchanged | deleted | delete locally | + | new | — | upload as new | + | — | new | download as new | + +5. **Applies** the actions, folders before their contents, a few at a time. +6. **Settles.** The baseline is updated from what actually happened. A failed local delete or + conflict copy keeps the old baseline. + +Safety rules: + +- **Wipe guard.** A pass in which every previously synced file has vanished locally is refused + instead of trashing the whole folder on Drive. It applies to any non-empty baseline. +- **Open for writing.** Files that some process holds open for writing (`/proc/*/fd`, + `open_for_write_set`) are left out of the pass and picked up after they are closed. +- **Ignore rules.** `.pdfsignore` (or `.protonignore`) at the folder root plus `ignore_patterns` + from `config.json`, in gitignore syntax. Ignoring never deletes. + +### Moves between locations + +My files and each online-only folder are separate FUSE sessions, and a mirrored folder has none, +so the kernel answers `rename(2)` between them with `EXDEV` and `mv` copies and deletes. On Drive +the same move is one `move_node`. `Request::Move` with two absolute paths (`pdfs move`, the app) +is resolved against every location by `Core::move_between` (`relocate.rs`): + +- **Nothing that exists only on this disk may be lost.** A mounted source is refused while it or + anything under it is queued or open for writing. A mirrored source is refused unless every file + under it matches its baseline (`mirror_subtree_unsynced`); ignored files and symlinks count as + unsynced. +- **Mirror to mirror** renames the local copy with `RENAME_NOREPLACE` and moves its baseline rows + (`sync_entries_move`), so neither folder's next pass has work. A failed `move_node` undoes the + rename. +- **Passes are held off.** Every mirrored folder involved is locked by ascending id, for at most + five seconds. +- **No downloads for what is cached.** A mirrored destination that has to download the node first + copies cached blocks into place (`ContentCache::copy_cached_to`). + +### Devices and the machine profile + +Synced folders live under a **device**, a per-computer root in Drive. `Core::ensure_device` picks +this machine's device: the adopted `device_uid` from `config.json` when set, else a Linux device +named like the hostname, else a new one. An adopted uid that no longer exists is reported, never +replaced silently. + +The local database does not survive a lost machine, so the daemon backs up its arrangement to +`/.proton-drive-linux/profile.json` (`pdfs_core::profile`): each synced folder's +remote uid, local path and mode, the pins, and the ignore patterns. It is uploaded like any other +file, so it is end-to-end encrypted. `Core::touch_profile` coalesces bursts of changes into one +upload. `PROFILE_VERSION` guards the format: a profile from a newer client is refused, not applied +in part. + +Restore lists the device's folders, proposes a local path from the profile, and on confirmation +adds a synced folder bound to the remote uid. The download is an ordinary pass against an empty +baseline. The user side is in [RECOVERY.md](RECOVERY.md). + +--- + +## 8. Control protocol + +Front ends talk to the daemon over a Unix socket, `control.sock` in the state directory +(`pdfs_core::control`). + +- **Framing.** One JSON `Request` per line, one JSON `Response` per line. +- **Limits.** A request is at most 1 MiB, must end with a newline, and must arrive within 10 s. At + most 64 handlers run at once. +- **Client timeouts.** 2 s to write a request; 120 s to read the answer; 5 s for a status poll, + which is what the app shows as "Not responding". +- **Errors.** `Response::Error { message, kind }`. The `kind` is machine-readable, and + `pdfs --json` passes it through. +- **Compatibility.** After an upgrade the old daemon runs until the service restarts, so a front + end must cope with a daemon that does not know a newer request. The daemon answers such a + request with an error instead of closing the connection. New fields get `#[serde(default)]`; + new behaviour gets a new request rather than an optional field an old daemon would ignore. + +### Push events + +The app and the tray send `Request::Subscribe` once and keep the connection open. The daemon then +streams one `Event` per line: + +| Event | Sent when | +|---|---| +| `Status` | On subscribe, and whenever a status field changes | +| `Transfers` | On subscribe, and whenever the transfer list changes | +| `Changed { topics }` | Something a page shows changed. Topics: `files`, `photos`, `trash`, `shares`, `devices`, `locations`, `queue`, `conflicts`, `activity` | +| `Heartbeat` | After 10 s of silence | + +Code that changes state publishes topics to the `EventHub` (`events.rs`); the control handler +lists the topics each mutating request touches. Changes within 150 ms are coalesced. A new +subscriber is told every topic changed. At most 8 subscriptions are served at once. + +`pdfs_core::control::follow` reconnects across daemon restarts with backoff and treats 25 s +without a line as a stuck daemon. + +### Search + +`SearchV2` carries the query, a limit, the sources (`Drive`, `Local`) and a content kind. The +daemon searches Drive metadata and its index of the home directory (`pdfs-localindex`), and scores +both with `pdfs_core::search::relevance_score`, so the prompt can merge them into one list. + +--- + +## 9. Threads + +| Thread | Does | +|---|---| +| FUSE dispatch loop, one per session | Reads kernel requests. Cheap ones are answered inline; the rest go to the worker pool | +| FUSE workers (`FUSE_WORKERS = 11`) | Handlers that may touch the network. 3 serve metadata only (`lookup`, `readdir`, namespace changes), so a listing never waits behind block downloads; 8 prefer transfers and help with metadata when idle | +| `pdfs-drain-0` … `pdfs-drain-15` | The drain (§5). Worker 0 also runs the queue's idle chores | +| `pdfs-control` | Accepts control connections; at most 64 handlers | +| `pdfs-events` | Publishes status, transfer and topic changes to subscribers | +| `pdfs-sync`, `pdfs-sync-poll` | Mirror passes and the remote poll (§7) | +| Tokio runtime | Remote calls, and the Drive and photos event streams (§6) | +| `pdfs-online-probe` | Probes Drive while offline | +| `pdfs-supervisor` | Logs workers that hold one job for minutes, a queue that stops draining, and control requests that never return; samples memory; pings the systemd watchdog only after a round trip over the control socket succeeds (`WatchdogSec=120`) | +| `pdfs-conflict-sweep` | Optional. After 30 s, then every 5 minutes, finds `(sync-conflict …)` copies identical (size and SHA-1) to their original. `conflict_sweep` decides: `off`, `report` (default) or `enforce`, which trashes them after re-checking each one | +| Smaller helpers | `pdfs-localindex` (home index for search), `pdfs-pause` (ends a timed pause), `pdfs-similar` (similar photos, on request), `pdfs-diagnostics` | + +Long-running loops wait on one shared `Shutdown` signal instead of sleeping, so a stop joins them +instead of leaving them behind (`docs/BUGS.md` B44). `pdfs diagnostics` reports what each worker +is doing. + +--- + +## 10. Threat model: what this client writes to disk in plaintext + +Proton Drive is end-to-end encrypted: the server never holds the keys to your content. That +property ends at this daemon. Serving a file through a POSIX filesystem means producing +plaintext, and serving it quickly means keeping some of it. This section states exactly what lands +on disk. + +**The short version: the cache and state directories hold decrypted content and decrypted +metadata. This client assumes the disk under them is encrypted (LUKS, or an encrypted home).** On +an unencrypted disk, someone with the powered-off machine can read cached files and the names and +structure of your whole Drive without your password. + +### 10.1 Decrypted content + +Everything under `$XDG_CACHE_HOME/proton-drive-linux/content/` is plaintext: + +| Path | Holds | Kept until | +|---|---|---| +| top level | Whole files: available offline, or opened whole | Evicted, or unpinned | +| `blocks/` | Blocks of partly read files | Evicted | +| `thumbs/` | Thumbnails and previews | Evicted | +| `scratch/` | Files open for writing | Released, or rescued at the next start | +| `staging/` | Released writes waiting to upload | **The upload lands** | +| `recovery/` | Fsynced writes rescued after an unclean stop | **Queued into `staging/`** | + +`staging/` and `recovery/` are not copies of something Drive has. They hold content that may exist +nowhere else yet, so they are never cleared at start or by a cache purge. They are both the most +sensitive data on disk and the data that must not be deleted to free space. + +### 10.2 Decrypted metadata + +`$XDG_STATE_HOME/proton-drive-linux/cache.db` is a plain SQLite database. It holds decrypted node +names, the folder tree, sizes, timestamps, a trigram index over names, the activity log, the +photos timeline and the queue. It is not evicted and not budgeted. Clearing the cache does not +touch it. + +`sdk_cache.db` holds the SDK's decrypted node metadata, encrypted at rest with a key from the +mailbox password (§12). + +### 10.3 What is not written in plaintext + +Credentials. The session (access and refresh tokens and the key material needed to resume) is +stored only in the system keyring through the Secret Service (`auth.rs`). `config.json` holds +settings and uids, no secrets. + +The control socket is a credential of another kind: anything that can connect to `control.sock` +can drive the daemon (list and read, upload, trash, create public links) without the keyring. See +§10.5. + +### 10.4 Memory, swap and the page cache + +Two exposures this client does not mitigate: + +- **Swap.** Keys and decrypted buffers live in ordinary heap memory. Nothing calls `mlock(2)`, so + they can be paged out. Use encrypted swap. +- **Kernel page cache.** Plaintext returned through FUSE is cached by the kernel like any file + data, and can be swapped. Avoiding that would mean `direct_io` on every read and losing the + kernel's read-ahead and caching. The client takes the performance. + +### 10.5 Ownership and file modes + +At every start, `AppDirs::ensure` checks that the config, state and cache directories are real +directories owned by the user, refuses symlinks and other owners, and sets mode `0700`. The control +socket is set to `0600` right after it is bound (`restrict_socket`); if that fails, the daemon +does not serve. + +| Artifact | Why it is restricted | +|---|---| +| `content/` | Another local user could read cached file content | +| `cache.db` | Another local user could read the names and structure of your Drive | +| `control.sock` | **Another local user could drive the daemon** with your session | + +`config.json` is written to a restricted temporary file, synced, renamed into place, and the +directory synced. A config file that cannot be parsed is reported and left alone, never replaced +with defaults. + +### 10.6 What follows + +- Treat the cache and state directories as being as sensitive as your Drive. +- Restored or copied state must stay owned by the user, or the daemon refuses to start. +- `pdfs cache clear` removes cached content, not `cache.db`, and never removes unsent `staging/` + or `recovery/` content. + +[RECOVERY.md](RECOVERY.md) covers what a lost machine means for this data and what to revoke. + +--- + +## 11. Sign-in and CAPTCHA + +Proton may answer a sign-in from an unfamiliar network or a VPN with a human verification +challenge (API error 9001). ```mermaid sequenceDiagram - autonumber - actor User as User (GUI) - participant Core as pdfs-core (Auth) - participant UI as pdfs-gui (Login Page) - participant Web as WebKitWebView Dialog - participant API as Proton API Server + actor User + participant UI as pdfs-app + participant Auth as pdfs-core auth + participant Web as WebKitGTK dialog + participant API as Proton API - User->>UI: Enter credentials & click Sign In - UI->>Core: login_interactive() - Core->>API: auth/v4 (Initial SRP Handshake) - API-->>Core: HTTP 422 (Error 9001: CAPTCHA Challenge URL) - Core->>UI: Error::HumanVerificationRequired(hv) - UI->>UI: Block login thread, dispatch to GTK main loop - UI->>Web: Create dialog & load verification URL - Note over User, Web: User completes CAPTCHA in embedded WebView - Web->>UI: window.postMessage(HUMAN_VERIFICATION_SUCCESS) - UI->>UI: Extract token (with double-serialization safety) - UI->>UI: Close dialog & send token to blocked login thread - UI->>Core: login_verified(with verification token) - Core->>API: auth/v4 (SRP retry with x-pm-human-verification-token) - API-->>Core: Returns session tokens - Core-->>UI: Login Successful + User->>UI: email and password + UI->>Auth: login_interactive() + Auth->>API: SRP handshake + API-->>Auth: 9001, challenge URL + Auth-->>UI: HumanVerificationRequired + UI->>Web: load challenge + User->>Web: solve it + Web-->>UI: token (postMessage) + UI->>Auth: login_verified(token) + Auth->>API: new SRP handshake with the token + API-->>Auth: session ``` -### 9.2 Key Technical Design Decisions - -1. **Weak Reference UI Binding:** To prevent memory leaks and strong reference cycles between the parent dialog, the child `WebKitWebView`, the script message manager, and the connection callback, the dialog is downgraded to a `WeakRef` inside the callback: - ```rust - let dlg_weak = dialog.downgrade(); - content.connect_script_message_received(Some("hv"), move |_, value| { - // ... - if let Some(dlg) = dlg_weak.upgrade() { - dlg.close(); - } - }); - ``` -2. **Double-Serialization Tolerance:** The JavaScript message listener forwards event data as a JSON string to the native handler. Since the underlying page may post either JS objects or pre-serialized JSON strings, the Rust side performs dual-phase parsing: - ```rust - let mut value = serde_json::from_str(raw).ok()?; - if let Some(inner) = value.as_str() { - if let Ok(parsed) = serde_json::from_str(inner) { - value = parsed; - } - } - ``` - This ensures compatibility with all versions of Proton's client verification scripts. -3. **SRP Handshake Reset:** Because a gated login burns the SRP handshake on the API side, the client cannot simply resume the previous request. Instead, `auth::login_interactive` restarts the SRP process from scratch with the verification credentials attached, keeping the complex handshake details isolated from the front-end. -4. **CLI Fallback:** Since the CLI has no native web browser engine, hitting the CAPTCHA gate fails immediately with a user-friendly message directing the user to sign in once via the GUI (`pdfs-app`) to persist the authenticated session keys to the system keyring. +- The challenge used up the first SRP handshake, so the retry starts a new one with the token + attached. +- The page may post the token as an object or as a JSON string; both are accepted. +- The CLI has no web view. It fails with a message to sign in once with `pdfs-app`; the stored + session then works for the CLI and the service. --- -## 10. Feature Design Notes +## 12. Feature notes -Notes on individual features whose code is shaped by a constraint that is not obvious from the -code alone. +Features whose code is shaped by a constraint that the code alone does not explain. -### File Version History -**Files**: [`revisions.rs`](../crates/pdfs-fuse/src/revisions.rs), [`control.rs`](../crates/pdfs-core/src/control.rs), [`versions_dialog.rs`](../crates/pdfs-gui/src/app/widgets/versions_dialog.rs) +### Version history -Proton Drive keeps every revision a client committed; the daemon only ever addressed the active one, so a file overwritten by a sync pass could be recovered only from whatever the local `recovery/` directory happened to hold. +`revisions.rs`, `versions_dialog.rs`. `ListRevisions`, `RestoreRevision`, `DeleteRevision` and +`SaveRevisionAs`, each with a `…ByUid` twin for nodes no mount can name. -The control protocol gained `ListRevisions` / `RestoreRevision` / `DeleteRevision` / `SaveRevisionAs` (each with a `…ByUid` twin for nodes the primary mount cannot name), the CLI gained `pdfs versions list|restore|save|rm`, and the browser's details pane gained a **Versions** button opening a per-file dialog. +- **A restore runs on the server, later.** Drive answers 202 and swaps the active revision in the + background. No content is sent and nothing is queued; the daemon evicts the file's cached + blocks and readers, and the app promises the request, not the result. +- **The active revision cannot be deleted.** Checked locally, so the user gets a sentence instead + of an API code. +- **Saving a version never overwrites.** An existing destination is refused, and a partial file is + removed when the download fails. -Three properties that shape the code: +### Photos -- **A restore is server-side and asynchronous.** No content crosses the wire and nothing enters the drain queue; the server answers 202 and swaps the active revision in the background, so the daemon evicts the file's cached blocks and open readers rather than describing the new state, and the UI wording promises the request, not the result. -- **The active revision cannot be deleted.** `Core::delete_revision_for_uid` checks that locally so the user gets a sentence rather than an API code, and the dialog gives the current row no delete button. -- **Saving a version never overwrites.** `SaveRevisionAs` refuses an existing destination and removes a partial file if the download fails — a half-written export looks identical to a good one to every tool that opens it afterwards. +`pdfs-fuse/src/photos.rs`, `pdfs-core/src/db/photos.rs`, `pdfs-gui/src/app/pages/photos.rs`. -### Photo Favorites -**Files**: [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs) +- **Favorites** use the SDK's photo tags. Favoriting a photo that is not on the account's own + photos volume (shared, or only in an album) is refused: it would need re-encryption the SDK does + not offer. +- **RAW + JPEG** pairs share a `group_key`. The server's relation (`main_photo_uid`, + `related_photo_uids`) comes first; otherwise the same capture day and name stem with one member + raw. The tile shows the non-raw member. Trashing a tile trashes the whole group. Albums are + never grouped. +- **The gallery** lays each day out in justified rows (`justify_rows`), so every tile has its + photo's shape. +- **Delete** is by uid (`Request::TrashNodes`), because the photos volume is not in the mount. The + rows are dropped at once, and the app offers Undo. -The gallery can mark and filter favorites, through the SDK's photo-tag API (`update_photos`). Schema **v20** adds `photos.favorite`; the flag is learned in the timeline enrichment pass that already resolves each photo's name and media type, and is kept across a refresh that could not resolve a photo — the same learned-and-kept rule as `media_type`. The lightbox carries a star toggle, the Photos header a favorites filter, and the CLI has `pdfs favorite [--remove]` plus `pdfs photos --favorites`. +### SDK entity cache -Favouriting a photo that is not on this account's own photos volume (shared with us, or album-only) needs it re-encrypted for our timeline root, which the SDK does not implement; the daemon surfaces that as an error rather than silently doing nothing. +`sdkcache.rs`. The SDK's cache of decrypted node metadata is stored in `sdk_cache.db`, so a restart +does not fetch and decrypt the tree again. -### Justified Gallery, Gallery Delete -**Files**: [`photos.rs`](../crates/pdfs-gui/src/app/pages/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs) +- **Its own file**, not `cache.db`: the traffic is frequent, small and can be rebuilt, and it needs + no migrations. It may be deleted while the daemon is stopped. +- **Encrypted at rest** with the SDK's `EncryptedCacheRepository`, keyed by the mailbox password. + After a password change it reads as empty. +- **Staleness** is handled by the event stream (§6). When the cursor has to be seeded (first + mount, or a lost cursor), the store is cleared. Signing out deletes it. -The gallery lays each day out in justified rows (`justify_rows` / `plan_rows` / `fit_row`): photos are taken in capture order until their summed aspect ratio no longer fits the target row height, and the row is then scaled so it ends exactly on the content width. A tile is therefore its own photo's shape, and `ContentFit::Contain` has nothing left to crop. The last row of a day is left at the target height rather than stretched, because a day holding two photos is a short day, not a layout fault. +### Batched calls -- **Ratios that are not known yet.** `PhotoItem::ratio` is persisted, but a photo that has never been decoded has none. Those are laid out square and remembered in `assumed_ratios`; when a decode proves the real shape the day re-flows on the existing debounce timer, so a screenful of decodes costs one re-flow rather than one per photo. Ratios are clamped to 0.4–3.0 so a single panorama cannot decide what a row looks like. -- **Thumbnails survive recycling.** A reply for a tile that has already been re-bound still lands in the texture cache, which is an LRU of 1500 textures keyed by uid, and the rows just outside the realised range are prefetched in the scroll direction. -- **Delete is uid-addressed.** The photos volume is not in the FUSE mount, so the path-based `Request::Delete` cannot reach it. `Request::TrashNodes { uids }` trashes through the SDK and answers `Response::Trashed { trashed, failed }`; the daemon then drops those rows with `Db::photos_delete` (album membership included) so the gallery does not wait for a timeline refresh. The GUI removes the tiles optimistically with an Undo toast and puts back anything the server refused. -- **Remote deletions arrive on their own.** The Drive volume and the photos volume have separate event streams, so `run_photos_event_sync` polls the photos volume with its own cursor (`photos_event_cursor` in the state table). Trash and delete events remove rows; anything else only invalidates freshness, because there is no inode space to converge on this volume. Together with a 60 s `TIMELINE_TTL` and a `RefreshScope::Photos` that now awaits the refresh, a photo deleted on a phone leaves the grid in about ten seconds. +`batch.rs`. `trash_nodes`, `restore_nodes` and `delete_nodes` report one outcome per node; +`batch::into_unit` turns a single-node call back into a `Result`. A restore first adds every +trashed ancestor and descendant of what was asked for, from the trash listing (`expand_restore`), +and sends them shallowest first, because Drive cannot restore a node into a parent that is still +trashed. Restore and permanent delete from the Trash page use the streaming variants and update +local state per node as each batch lands. -### RAW + JPEG Grouping -**Files**: [`photos.rs`](../crates/pdfs-core/src/db/photos.rs), [`migrations.rs`](../crates/pdfs-core/src/db/migrations.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs) +--- -Schema **v30** adds `content_hash`, `main_uid` and an indexed `group_key` to `photos`. The first two come from the server's `PhotoProperties` and are filled by the enrichment pass in `refresh_timeline` that already resolves each photo's name, media type and favorite tag; `group_key` is computed in `photos_replace`, where the whole timeline is in hand. +## 13. Where it is going -- **Precedence.** The server relation first (`main_photo_uid` and `related_photo_uids`, read from whichever end resolves — the two can land in different enrichment chunks); then same capture day, same case-insensitive name stem, and one member raw while the other is not; otherwise the photo is its own group. The union-find pass is over the timeline in memory, so it costs one pass per refresh rather than a query per photo. -- **Representative.** The non-raw member when there is one — a JPEG decodes in milliseconds and is what the person expects to see. `group_key` holds that member's uid, so "is this the tile" is `uid = group_key`, an index scan. -- **What is grouped and what is not.** `photos_page`, `photos_months` and `photos_counts` count groups, except on the Raw tab, which lists files. Album pages are never grouped: an album is a list someone made. The relation is learned-and-kept like `media_type`, so a refresh that could not resolve a node does not break a group up. -- **Deleting.** `Core::trash_photos` expands each uid to its group before trashing, because the grid shows the shot as one tile — leaving the RAW behind would put the photo back on the Raw tab and nowhere else. The GUI's confirmation names the file count. -- **The other files.** `Request::PhotoGroup { uid }` answers with the group's members as ordinary `PhotoItem`s, which is what the lightbox's switch steps through. +3.0.0 made the mount local-first. What is still split: -### Persistent SDK Entity Cache -**Files**: [`sdkcache.rs`](../crates/pdfs-core/src/sdkcache.rs), [`auth.rs`](../crates/pdfs-core/src/auth.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs) +- **The old online path** in `filesystem.rs` and the `local_first` switch go in the next release. +- **Remote changes** are applied by the event code and by several invalidation helpers. One + applier for everything that comes from Drive is the next step. +- **Mirrored folders** still have their own engine (§7). They move onto the local tree and the + queue after that. -The SDK's Drive entity cache (decrypted node metadata: name, size, parent, signing share) defaulted to memory, so every daemon restart re-fetched and re-decrypted the tree the previous run had already walked. It is now backed by SQLite. - -- **Its own file** (`sdk_cache.db` in the state directory), not `cache.db`: the daemon's `Db` is one `Mutex` shared by every FUSE thread and the control socket, and this traffic is frequent, small and entirely reconstructible. A separate file also means no schema migration — the store can be deleted at any time. -- **Encrypted at rest** by wrapping it in the SDK's `EncryptedCacheRepository`, keyed by the mailbox password. A password change reads as a cold cache (the SDK treats an undecryptable entry as a miss and clears the store), not as an error. -- **Staleness** is closed by the event loop, which already replays from a persisted cursor and calls `invalidate_caches_for_event` for every event — including those raised while the daemon was down. The one case with no trail is a *seeded* cursor (first-ever mount, or a lost cursor), where the seed path now clears the store instead of trusting it. `auth::logout` removes the file. - -Needed a small additive SDK change (`ProtonDriveClient::with_entity_repository`, 0.5.1): `with_entity_cache` is a constructor and so could not be combined with `with_key_salts`, which the daemon requires. - -### Per-node Batch Outcomes -**File**: [`batch.rs`](../crates/pdfs-core/src/batch.rs) - -SDK 0.5.0 made `trash_nodes` / `restore_nodes` / `delete_nodes` report one outcome per node instead of failing the whole call (mirroring upstream's streamed `NodeActionResult`). `pdfs_core::batch::into_unit` collapses the single-node calls back to a `Result`; `batch::split` handles a collected batch. - -A restore is expanded before it is sent (`expand_restore`, `pdfs-fuse/src/lib.rs`): the persisted trash listing carries each row's `parent_uid` (schema 29), so a restore takes the connected piece of the trashed tree — every trashed descendant of what was asked for, and every trashed ancestor above it — and sends it shallowest wave first, because the server cannot put a node back under a parent that is still trashed. A uid the listing does not know about (it is materialised in chunks, and can be stale) is restored on its own rather than dropped. - -The trash view's restore and permanent-delete use the SDK's **streaming** variants (`restore_nodes_streaming` / `delete_nodes_streaming`) and apply local state per node as each batch lands rather than after the last one. For a permanent delete — which is irreversible — that means a daemon interrupted mid-batch has forgotten exactly the nodes the server destroyed, no more and no fewer. Both report per-node failures and count only what actually succeeded; only a batch where nothing succeeded is an error. - -### Moves Between Locations -**Files**: [`relocate.rs`](../crates/pdfs-fuse/src/relocate.rs), [`control.rs`](../crates/pdfs-fuse/src/control.rs), [`browser.rs`](../crates/pdfs-gui/src/app/pages/browser.rs) - -The primary mount and every on-demand folder are separate FUSE sessions, and a mirror folder has no session at all. The kernel answers `rename(2)` between two mounts with `EXDEV` before the daemon hears of it, so `mv` copies and deletes. On Proton Drive the same move is one `move_node`, because My files and this device's folders are on the main volume. - -`Request::Move` with two absolute paths is therefore resolved by `Core::move_between` against every location: a live mount through `rooted_at`, otherwise the mirror folder with the longest `local_path` prefix, whose uids come from the `sync_entry` baseline. Two paths in one inode space still go to `move_to`. Relative paths keep their old meaning. - -- **Nothing that exists only on this disk may be lost.** A mounted source is refused while it or anything under it has a queued op or a file open for writing. A mirror source is refused unless every file under it matches its baseline and every baseline row still has its file (`mirror_subtree_unsynced`); ignored names and symlinks count as unsynced, because the local copy is removed after the move. -- **Mirror to mirror renames along.** The local copy is renamed first, with `RENAME_NOREPLACE` so nothing that appeared at the destination meanwhile is overwritten, and its baseline rows are moved with `sync_entries_move`, so neither folder's next pass has anything to do. If `move_node` fails, the rename is undone. `EXDEV` between two filesystems falls back to the check above, then drops the source copy. -- **A dropped mirror copy is checked twice.** The check runs again just before the source copy is deleted. A file written into it during the move keeps the whole copy; its baseline is gone, so the next pass uploads it as new. -- **Sync passes are held off.** Every mirror folder involved is locked through `sync_lock`, by ascending id, with a five-second bound. A folder that is still busy is reported rather than waited on. -- **The local side is levelled afterwards.** A mounted source forgets the node in every inode space and notifies the kernel. A mounted destination has its listing cleared. A mirror destination that received nothing locally gets a reconcile, which downloads the node. That download first asks `ContentCache::copy_cached_to` for the file: a whole cached blob, or every block of it, is copied into place instead of fetched, so a move out of an on-demand folder costs no traffic for what was already opened there. - -### Open-for-Write Deferral for Mirror Sync -**File**: [`sync.rs`](../crates/pdfs-fuse/src/sync.rs) - -The mirror folder sync engine now defers uploading any file that is currently held open for writing by another process, matching the guarantee the FUSE mount path already provides (where uploads are deferred until `close(fd)`). - -**Problem**: Previously, the mirror sync path relied solely on a 2-second trailing-edge debounce (with a 30-second ceiling) to avoid uploading files mid-write. This was insufficient for slow continuous writers (e.g., database dumps, large exports, or editors that keep files open for extended periods), which could have their incomplete state uploaded as a real revision. - -**Solution**: Before each reconcile pass, the sync engine scans `/proc/*/fd` once to build a set of canonical paths within the sync root that any process holds open for writing (`O_WRONLY` or `O_RDWR`). Files in this set are: - -- **Kept in the local walk** — so they are not misclassified as deletions -- **Treated as unchanged** — so no upload is queued for them -- **Counted as `deferred`** — so the activity summary reports them (e.g., "3 uploaded, 1 deferred (open for write)") -- **Picked up on the next pass** — after the writer has closed the file - -**How it works**: -1. `open_for_write_set(root)` reads `/proc/*/fd` → `readlink` for each fd → prefix-checks against the sync root → reads `/proc//fdinfo/` to check the `flags:` line's low two bits (O_ACCMODE) -2. `walk_local` sets `LocalItem.open_for_write = true` for matching files -3. Both `do_reconcile` and `push_pass` skip upload classification for flagged files -4. `Outcome.deferred` tracks the count for the activity summary - -**Performance**: The `/proc` scan completes in low single-digit milliseconds on a typical desktop. Only fds whose `readlink` target falls under the sync root incur the `fdinfo` read. - -**Tests added**: 8 unit tests covering `is_write_mode` parsing of various fdinfo flag combinations, and `Outcome` summary formatting with deferred counts. - -| Sync Path | Open-for-write protection | Mechanism | -|---|---|---| -| FUSE mount | Yes | Upload deferred until `close(fd)` | -| Mirror folders | Yes | `/proc/*/fd` scan per reconcile pass | +[ROADMAP.md](ROADMAP.md) tracks these and the rest of the planned work. diff --git a/docs/BUGS.md b/docs/BUGS.md index c3e1458..ce3c3b4 100644 --- a/docs/BUGS.md +++ b/docs/BUGS.md @@ -255,7 +255,7 @@ B143 does, after a restart too. Sim test **Status:** Fixed. **Found:** 2026-10-05, checking that every name collision and conflict the drain resolves is -reported (`docs/MILESTONE-3.0.0.md` §6.2). Each one is in the activity log, and the app words +reported (sync issues, `docs/ARCHITECTURE.md` §5). Each one is in the activity log, and the app words each in the user's language but one. **Where:** `detail_label` in `crates/pdfs-gui/src/app/pages/activity_text.rs`; the daemon diff --git a/docs/MILESTONE-3.0.0-STATUS.md b/docs/MILESTONE-3.0.0-STATUS.md deleted file mode 100644 index 2e0c837..0000000 --- a/docs/MILESTONE-3.0.0-STATUS.md +++ /dev/null @@ -1,182 +0,0 @@ -# 3.0.0: what is done and what is left - -Tracks `docs/MILESTONE-3.0.0.md`. Update it with every commit that moves a phase. - -## Done - -- **Phase 0**: fake Drive, simulation runs, CI job (2.9.0). -- **Phase 1**: queued-op preconditions and own sealed revisions in the database (2.9.0). -- **Phase 2**: local ids, inode = local id, one tree for every mount (2.10.0). -- **Phase 3, first part**: mkdir, create, rename, unlink and rmdir on the mount are recorded - locally and sent from the queue; `"local_first": false` brings back the old path (5e0f30c). -- Fixes from the simulation and account runs since then: B125, B127, B130 and B141 to B193. -- **Sync issues, first cut** (§6.2): refusals sorted into issues, shown at once in the app, - tray and CLI; Export button and `pdfs sync export`; `pdfs sync issues`; Discard button and - `pdfs sync discard`; a warning badge on the node in the file browser. - -## Phase 3: left for 3.0.0 - -1. **Sync issues (§6.2)**: first cut done. - - [x] The drain sorts a refusal from Drive into an issue kind (quota, access, missing, - limit, rejected) and stores it on the queued op (schema 38). - - [x] An op with an issue counts as "needs attention" at once, not after six failures. - - [x] The queue listing carries the issue kind and whether its content can be exported. - - [x] Export a queued file's content (control request, `pdfs sync export`). - - [x] `pdfs sync issues`, and the issue shown in `pdfs sync queue`. - - [x] The app's Sync page shows each issue with what the user can do; Export and Retry - buttons; translated in every catalog. - - [x] A name collision lands under a conflict name (create, rename, move, re-homed create) - and goes in the activity log, as does a queued edit kept as a conflict copy. The app - words each one in the user's language (the last one since B181). - - [x] Discard an issue on purpose (after export, or for a change with no content), and - undo the local change it stood for: a node made here goes, a node Drive has goes back - to what Drive has. Control request, `pdfs sync discard`, Discard button with a - confirmation; translated in every catalog. - - [x] Show the issue on the node itself: a folder listing carries each node's issue, and - the file browser shows a warning badge with what it means. The listing is published - again when the set of issues changes. -2. **Planner and executor (§5.4, §5.5)**: ops keyed by local id; remove `mint_local_uid`, - `finish_create` and `adopt_real_uid`; a landing create sets `remote_uid` on its row. - - [x] Queued ops carry `lid` and `parent_lid` (schema 39). The claim reads the parent's uid - through `parent_lid`, and a landing folder rewrites the ops inside it by local id. - - [x] Node rows link to their parent by lid (`nodes.parent_lid`, schema 40), and a restart - places each node under its parent's row as it is now. - - [x] Queries over children, subtrees and ancestors by `parent_lid`, and no rewrite of the - children's `parent_uid` when a folder lands (R3), nor of the ops made inside it (R2). - - [x] No more placeholders: a node made here goes by its local id until it lands. In four - parts: - - [x] 6a: a node made here goes by `local~`, a function of its row, instead of a - minted `local~-`; schema 41 rewrites the old placeholders (nodes, ops, pins). - - [x] 6b: an op is found by its node's local id under either uid, the claim keeps one op - per local id on the wire, and the drain sends the uid the row has now; - `remap_local_uid` is gone. - - [x] 6c: children, subtrees, ancestors and ops' parents by lid, so a landing create - rewrites no other row or op; what is read for a uid names the parent as its row has it - now. The row keeps its stand-in in `nodes.uid` rather than `NULL`: it is as stable, - and `NULL` would have changed some 40 node queries for no change in behaviour. Pins - stay keyed by uid and move with the row in the landing transaction (B148). - - [x] 6d: a node written under its stand-in after it landed updates its landed row - instead of bringing the placeholder row back, so a write that finds its create - landed takes the uid from the row and no longer polls the tree for up to 2 s - (`landed_uid` is gone). In memory, `pending`, the uploads and the tree stay keyed by - the uid Drive knows: ops are matched by lid in the database (6b), `hidden` and - `creating` only ever hold remote uids, and `landed_row_uid` is the one place at the - mount's edge where a stand-in a handle still holds becomes the real uid. - - [x] An executor that runs ops in dependency order, many at once: 16 drain workers instead - of 3, and a worker that lands an op wakes the idle ones, which slept through the ops it - made claimable. The claim already keeps the order: a create waits for its parent's - uid, a node's ops go one at a time. Ops stay blocking calls on worker threads: each - one spends its time waiting on Drive, so tasks would buy nothing threads do not. -3. **Speed (§7)**: run independent ops in parallel, batch trashes and moves; measure 1,000 files - and 100 folders on Wi-Fi against LAN (target: within 10 %). - - [x] The measurement: `a_thousand_files_drain_as_fast_on_wifi_as_on_lan` in `sim/daemon.rs`, - run with `PDFS_SIM_MEASURE=1`. It prints the times and asserts nothing yet. - - [x] Baseline on eb1c105, three drain threads: the syscalls take 3.7 s on LAN and 4.3 s on - Wi-Fi; the drain takes 138 s on LAN and 938 s on Wi-Fi, 6.8 times as long. - - [x] Sixteen workers: the syscalls take 5.2 s on LAN and 5.0 s on Wi-Fi, which meets the - target; the drain takes 47 s on LAN and 247 s on Wi-Fi. Most of a create on Wi-Fi is - its read-back waiting for Drive to list the new node (250 ms, then 1 s), so batching - (step 8) helps trashes and moves, not creates. - - [x] Batched trashes and moves: left for after 3.0.0 (see below). The target is met - without them, the drain waits on the creates' read-back, and splitting one answer into - many results is where the B15x to B17x races lived. -4. **Done-when checks**: - - [ ] Simulation passes on the Wi-Fi profile: `one_client_on_a_slow_link` failed in CI on - seeds 2 and 3. Two bugs (B127, B151) and a harness gap: a settle did not read the log, so - it could not explain a known bug's damage. All three fixed; not yet confirmed in CI. - Replays then found two races on the LAN profile too (B152, B153), both fixed. CI on - 7269cb5 failed LAN seed 9, a regression from the B153 fix (B154), and Wi-Fi seed 2, - a write whose answer was lost (B155). Both fixed. One slow-link run of seed 2 failed - once with a rename answering `ENOENT` as a folder move landed. Not reproduced since; - it may be B152. Replays of slow-link seed 4 failed two of six times on a file written - again after its create lost its answer (B156, fixed). With sixteen drain workers - (16fca10), seed 4 failed every replay on a write after an upload that was not read - back (B157, fixed). Seed 3 then failed four of ten runs on two renames of one file - landing out of order (B158, fixed). Flaky-link seed 4 failed one of ten runs on a - lookup that answered `EAGAIN` while the root was listed again after a restart (B159, - fixed). Slow-link seed 4 also had folders reach Drive 30 s late, after the - drain's idle poll (B160, fixed). Slow-link seed 3 failed on a file moved while its - create was on the wire, which the mount showed under its old name (B161, fixed). - Seed 4 failed on a folder removed and made again, whose create adopted the removed - folder (B162, fixed). CI on 903452a passed every profile. CI on f9b1638 failed - slow-link seed 2: a step that unlinks an open file, appends and reads it back hung - for 150 s. The B130 matcher did not fire, and CI kept no stacks; one replay passed. - It may be a kernel notice sent from a runtime worker by another path than B130's, - which a68dae9 (not yet pushed then) also takes off the runtime. CI now keeps a hung - run's stacks. - - [x] Known-bug list in `sim/run.rs` is empty: the run on 812eb3d (`PDFS_SIM_KNOWN=fail`, - twelve seeds) passed every profile, so the matchers are gone. That confirms B125 to - B133; B134 stays open with `local_first` off only, which the runs do not mount. The confirmation run (`PDFS_SIM_KNOWN=fail`) - on da013c2 failed `one_client_on_a_flaky_link` seeds 6 and 11 on two new bugs, both - after restarts: a write over a create never read back (B182) and a trash that forgot - what its create sent (B183). Replays with those fixed found a third: a create renamed - after it lost its answer (B184). All three fixed; no known-bug matcher fired. It was - stopped before `three_clients`. The rerun on 3073052 found a create that waited out - a retry after a write replaced its blob (B185, fixed), and a file deleted offline - while open that lost the bytes its handle read (B186, fixed; older than 3.0.0). - The run on 2baad67 found that a restart before such a file's trash went out queued - its kept bytes again, for good (B188, fixed), and changes that sat out their retry - after an outage only the drain had met (B189, fixed; older than 3.0.0), and a file - saved just before a stop whose release the stop dropped (B190, fixed; older too). The run - on 1436c5f found a file deleted after its failed create was written again left on Drive - (B191, fixed). Its test for B193 found a new empty file waiting out the idle poll (B192, - fixed), and seed 6 a deleted file's trash taking a file of the user's renamed to the name - its create was sent under (B193, fixed). - - [ ] A clean account run. The one on a6f36d5 (2026-10-04, LAN: 427 passed, 12 failed) - failed B113 and B121 on every on-demand mount, two mirror-to-mirror move cases, and - the journal check. B121 was a bug (B149, fixed); B113's case still expected creates - to reach Drive at once and now accepts queued ones. The move cases were the harness: - it retried every source after one was busy, and expected a refusal where a mirror to - mirror move carries the local copy along. The journal showed a conflict copy retrying - into a deleted folder (B150, fixed). Not confirmed yet. The run on 2.10.0 - (2026-10-04: 260 passed, 3 failed) failed a new file that read back empty (B163, - fixed), a 128-entry folder that listed 129 (B164, fixed), and its cleanup, which ran - while the daemon was restarting: the stop waited on a busy thread until systemd killed - it (B165, bounded). The next run on the working tree (2026-10-04: 259 passed, 4 - failed) found a wide folder dropping files whose create landed while it was listed - (B166, fixed), or listing one empty: the new file that read back empty again - (B166 too). Its cleanup found no control socket because it ran 1.5 s after the - last restart, before the new daemon had bound it: `pdfs status` reports that as not - running, which the run's queue wait takes for an empty queue (the script's side). - Looking into it found the start spending 1.7 s rebuilding listings (B167, fixed). - Its journal check caught a file removed with its folder while uploading being - sent again, to the root (B168, fixed), and a stop that let a mirror pass go on - until the runtime ended under it (B169, fixed). CI on 9bbc994 failed - `one_client_on_a_good_link`: a write opened as an upload landed took the revision - it replaced as its base, and the next write failed with `EIO` (B170, fixed). The - suite's seed 9 found a deleted file's trash waiting out its create's backoff (B171, - fixed). Run three at a time on 2502af6, seed 8 failed two of nine runs: a file renamed - while its upload was read back showed its old name again (B172, fixed). The loop - on d850265 failed seed 12 once in nine runs: a file deleted just as its create - landed stayed on Drive (B173, fixed). The suite on 378667f failed seed 6: a rename - in the same gap was refused with `EACCES` (B174, fixed). The loop on 378667f failed - seed 8: a file replaced by a rename in that gap stayed on Drive (B175, fixed), and - seed 12: a write closed as its file's create landed was dropped (B176, fixed). - The account run on 378667f: a folder listed as its create landed asked Drive by - its stand-in and answered `EIO` (B177, fixed). A folder trashed while sync was - paused kept a create below it as a trash: a paused drain worker had claimed it - (B178, fixed). A file read as its create landed read empty (B179, fixed). - The suite on 0f3c06f: a stop gave up on a drain worker that slept through its - wake-up (B180, fixed). The account run on 903452a (2026-10-04, Wi-Fi: 262 passed, - 1 failed) failed only its cleanup, which found no control socket: after the mirror - target's last restart the queue wait took the missing daemon for an empty queue - (the script's side, fixed in 6539c73). -5. **Release**: 2.10.0 is tagged; the 2.9.0 tag still waits for the user's go-ahead. - -## After 3.0.0 - -- Remove the old online path in `filesystem.rs` and the `local_first` switch (the release after - 3.0.0). -- **Phase 4**: one applier for remote changes. -- **Phase 5**: mirror folders on the same planner and executor. -- **Batched trashes and moves** (plan step 8). Not needed for the §7 target: the syscalls take - 5.2 s on LAN and 5.0 s on Wi-Fi. The drain's time goes to the creates' read-back (250 ms, - then 1 s each), which batching does not shorten. A batch call answers for many ops at once, - and splitting that answer back per op is the step the B15x to B17x races came from. First - measure `rm -r` of 1,000 files on Wi-Fi. If it is slow, drop the queued trashes of a trashed - folder's children before batching anything: trashing the folder takes them with it. -- **Any time**: `Size` and block geometry as types; a deadline on the remote calls outside - the drain (the drain's have one: `link::upload_deadline`, `Core::block_on_within`). -- Open questions in §12 of the milestone. -- `PLAN.md`: the acceptance script notices network drops and reports `interrupted` instead of - TIMEOUT. diff --git a/docs/README.md b/docs/README.md index 50bffd1..a691cdf 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,7 +5,7 @@ | Page | What it covers | |---|---| | [Installation](INSTALL.md) | Packages, building from source, first sign-in, updating and uninstalling | -| [User guide](USER_GUIDE.md) | My files, available offline, synced folders, conflicts, Photos, sharing, Trash and versions, Computers, the search launcher, the tray | +| [User guide](USER_GUIDE.md) | My files, available offline, synced folders, sync issues and conflicts, Photos, sharing, Trash and versions, Computers, the search launcher, the tray | | [Command-line reference](CLI.md) | Every `pdfs` command, grouped by task, and scripting with `--json` | | [Configuration](CONFIGURATION.md) | `config.json`, environment variables, file locations, the systemd service | | [Troubleshooting](TROUBLESHOOTING.md) | Diagnosing problems, common symptoms, starting over, reporting a bug | @@ -16,13 +16,12 @@ | Page | What it covers | |---|---| -| [Development guide](DEVELOPMENT.md) | Building, running and debugging from source; releasing | -| [Architecture](ARCHITECTURE.md) | Crates, read and write paths, sync engine, control protocol, threads, threat model | -| [Testing](TESTING.md) | Unit tests and the FUSE acceptance suite | +| [Development guide](DEVELOPMENT.md) | Building, running and debugging from source; common changes; releasing | +| [Architecture](ARCHITECTURE.md) | Crates, the local tree, read and write paths, the queue, the sync engine, the control protocol, threads, threat model | +| [Testing](TESTING.md) | Unit tests, simulation runs and the FUSE acceptance suite | | [Translating](TRANSLATING.md) | Adding and updating translations | | [Roadmap](ROADMAP.md) | Planned work and verification still owed | | [Bug ledger](BUGS.md) | Every tracked defect with its analysis and status | -| [UI/UX plan](UI_UX_PLAN.md) | Design audit and notes behind the 2.x desktop app | Also see [CONTRIBUTING.md](../CONTRIBUTING.md) and [SECURITY.md](../SECURITY.md) in the repository root. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f4c5b3d..4ca340e 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,8 +1,7 @@ # Roadmap -Planned work and open verification, as of 3.0.0. Individual defects and their status live in -[BUGS.md](BUGS.md), which is the authoritative ledger; this page groups what is left by theme. -Nothing here is a promise of a date. +Planned work and open verification after 3.0.0. Individual defects and their status are in +[BUGS.md](BUGS.md); this page groups what is left by theme. Nothing here has a date. - [Sync engine](#sync-engine) - [Data safety and recovery](#data-safety-and-recovery) @@ -14,38 +13,42 @@ Nothing here is a promise of a date. ## Sync engine 3.0.0 records changes made through the mount locally and sends them from the queue -([MILESTONE-3.0.0.md](MILESTONE-3.0.0.md)). What comes after it: +([ARCHITECTURE.md §4](ARCHITECTURE.md#4-write-path)). What comes next: | Item | State | Notes | |---|---|---| -| Remove the online path | Not started | The old way of sending a change before the syscall returns, and the `local_first` switch that brings it back, go in the release after 3.0.0 | -| One applier for remote changes | Not started | Phase 4 of the milestone: a remote tree, and remote changes applied through the same planner as local ones | -| Synced folders on the queue | Not started | Phase 5: synced folders sent by the same planner and executor as the mount | +| Remove the online path | Not started | The old way of sending a change inside the syscall, and the `local_first` switch that brings it back, go in the next release | +| One applier for remote changes | Not started | Everything that comes from Drive (events, listings) changes the tree in one place, which is also the only place that tells the kernel. Replaces today's invalidation helpers and in-memory overlays | +| Synced folders on the queue | Not started | Mirrored folders feed the local tree and use the same queue as the mount; `sync_entry` becomes the record of the last agreed state. Switching a folder's mode then moves no data except to evict or fetch content | | Batch trashes and moves | Deferred | See below | +| Provisional values as types | Not started | A size or block geometry that is not known yet as its own type, so it cannot reach the kernel as if it were known | +| Deadlines on every remote call | Partly done | The drain's calls and calls a syscall waits on have one; background calls outside the drain do not yet | -**Batching trashes and moves.** `trash_nodes` and `move_nodes_streaming` take up to 150 nodes per -call, but the queue still sends one node per call. It was left out of 3.0.0 on purpose: +**Batching trashes and moves.** `trash_nodes` and `move_nodes_streaming` take many nodes per call, +but the queue sends one node per call. It was left out of 3.0.0 on purpose: -- The speed target of the milestone (§7) is met without it. Making 1,000 files in 100 folders - takes 5.2 s on a LAN and 5.0 s on Wi-Fi. -- What the queue spends its time on is the creates. Each one waits for Drive to list the new node - before it counts as landed (250 ms, then 1 s), and batching does not help with that. -- Sending many ops in one call means splitting one answer back into many results. Races in - exactly that step were the largest class of bugs found for 3.0.0 (B15x to B17x). +- The 3.0.0 speed target is met without it. Making 1,000 files in 100 folders through the mount + takes 5.2 s on a LAN profile and 5.0 s on a Wi-Fi profile in the simulation + (`PDFS_SIM_MEASURE`, see [TESTING.md](TESTING.md#simulation-runs)). +- The drain spends its time on creates. Each waits for Drive to list the new node before it + counts as landed, and batching does not shorten that. +- One call answering for many ops has to be split back into results per op. Races in exactly that + step were the largest class of bugs found for 3.0.0 (B15x to B17x in [BUGS.md](BUGS.md)). Before building it, measure `rm -r` of a folder with 1,000 files on Wi-Fi. If that is slow, the -cheaper fix is to drop the queued trashes of the files in a folder when the folder itself is -trashed, since trashing the folder takes its contents with it. +cheaper fix is to drop the queued trashes of a trashed folder's contents: trashing the folder takes +them along. ## Data safety and recovery | Item | State | Notes | |---|---|---| -| Protect queued changes at shutdown | Not started | A systemd inhibitor lock while `staging/` holds unsent data, a warning when signing out, and a blocking `pdfs sync flush`. Today the queue is visible through `pdfs sync queue`, the tray and **Sync → Overview**, but nothing stops a shutdown (B66) | -| Restore pins from the machine profile | Not started | `profile.json` records pins; `sync restore` does not apply them yet | -| Profile backup health | Not started | A failed profile upload is a `WARN` in the log and nothing in the UI (B68) | -| Rotated credentials survive a keyring failure | Open | Surface and retry keyring write failures so a rotated single-use refresh token is not lost (B61) | -| Encrypt local state at rest | Not started | SQLCipher for `cache.db` and an AEAD for the content cache, keyed from the keyring. `sdk_cache.db` is already encrypted. See [ARCHITECTURE.md §8](ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext) | +| Protect queued changes at shutdown | Not started | A systemd inhibitor lock while `staging/` holds unsent data, a warning when signing out, and a `pdfs sync flush` that waits for the queue. Today the queue is visible in `pdfs sync queue`, the tray and **Sync → Overview**, but nothing stops a shutdown (B66) | +| Restore pins from the machine profile | Not started | `profile.json` records pins; `pdfs sync restore` does not apply them yet | +| Profile backup health | Not started | A failed profile upload is a `WARN` in the log and nothing in the app (B68) | +| Rotated credentials survive a keyring failure | Open | Report and retry keyring write failures, so a rotated single-use refresh token is not lost (B61) | +| Restore never picks a new device silently | Open | Ask when the machine's device is ambiguous after a loss of local state (B67) | +| Encrypt local state at rest | Not started | SQLCipher for `cache.db` and an AEAD for the content cache, keyed from the keyring. `sdk_cache.db` is already encrypted. See [ARCHITECTURE.md §10](ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext) | ## Features @@ -53,9 +56,10 @@ trashed, since trashing the folder takes its contents with it. |---|---|---| | File-manager integration | Not started | Context-menu actions (available offline, share link, versions) and sync badges for Nautilus, Dolphin and Thunar | | Terminal UI | Not started | `pdfs tui` with transfers, sync state and logs | -| Multiple accounts | Not started | Separate config, state and mountpoint per profile, one daemon each | +| Multiple accounts | Not started | Separate config, state and mountpoint per account, one daemon each | | Mount options | Not started | Read-only mount, `allow_other` | -| Symbolic links | Not started | Proton Drive has no link type; links would need a placeholder format. Until then, creating one fails with `EOPNOTSUPP` | +| Symbolic links | Not started | Proton Drive has no link type; links would need a placeholder format. Until then, `symlink` fails with `EPERM` | +| `fsync` that waits for Drive | Idea | `fsync` means "durable locally". A mount option or an xattr could offer "on Drive" without slowing the default | | Add someone else's photo to your album | Blocked | Needs `copyPhoto` in the SDK | | Photo name search | Not started | Photos search by date and filters only | | Automatic photo upload from a folder | Not started | | @@ -66,34 +70,38 @@ trashed, since trashing the folder takes its contents with it. | Item | Notes | |---|---| -| Saturating stream-ring accounting | `self.bytes -= …` in `crates/pdfs-fuse/src/reads.rs` should saturate so an accounting drift cannot wrap in release builds | +| Saturating stream-ring accounting | `self.bytes -= …` in `crates/pdfs-fuse/src/reads.rs` should saturate, so an accounting drift cannot wrap in release builds | | Block geometry learned one read late | The first read of a file plans on 4 MiB blocks; accepted as a bounded cost (B87) | -| Path-based commands under secondary mounts | `pin`, `rename`, `move`, `rm` and sharing cannot name a path inside an on-demand synced folder yet (B88) | -| Unicode normalization | NFC and NFD names are distinct, deliberately (B36) | -| Pipelined node enumeration | Faster cold listings of large folders | +| Path-based commands under online-only folders | `pin`, `unpin`, opening a file, uploads, sharing and versions cannot name a path inside an online-only synced folder yet. `rm`, `rename`, `mkdir` and `move` can (B88) | +| Unicode normalization | NFC and NFD names are distinct, on purpose (B36) | +| Pipelined node enumeration | Faster first listings of large folders | ## Verification still owed -Fixes marked *unverified* in [BUGS.md](BUGS.md) have unit or offline coverage but have not been -driven against a live account. The larger outstanding runs: +Fixes marked *unverified* in [BUGS.md](BUGS.md) have unit or simulation coverage but have not +been driven against a live account. The larger runs still owed: +- **A clean account run on 3.0.0.** The last account run before the release (2026-10-04, on Wi-Fi) + failed only its cleanup, on a script bug fixed since. No full run with `--journal-check` has + passed on the released code yet. - **Disaster-recovery drill.** Register a device, sync a folder, delete all local state, restore - with `pdfs sync restore` and compare the tree byte for byte. See [RECOVERY.md](RECOVERY.md). -- **Second-account sharing.** Shared trees on a foreign volume, accepting invitations, changing + with `pdfs sync restore`, and compare the tree byte for byte. See [RECOVERY.md](RECOVERY.md). +- **Second-account sharing.** Shared folders on a foreign volume, accepting invitations, changing roles, the viewer cases of `regression B34`, and an editor writing into someone else's share. -- **Fault injection.** Network loss mid-upload, a full disk during staging, and crash points - across the write publication path (after each fsync, rename and queue commit). +- **Fault injection on a real account.** Network loss mid-upload, a full disk during staging, and + crashes at each step of publishing a write. The simulation covers link loss and restarts against + the fake Drive, not a kill at an arbitrary point. +- **Acceptance runs across a network drop.** A case that spans an outage times out while the queue + replays, instead of being reported as interrupted. - **CAPTCHA sign-in** against a real verification request (B8). -- **Managed live mode matrix** before each release: - `scripts/fuse-acceptance.sh --managed-live `. See [TESTING.md](TESTING.md). -- **The overnight hang** (B91) is not diagnosed yet. The supervisor and `pdfs diagnostics` exist to - catch the next occurrence. +- **The overnight hang** (B91) is not diagnosed. The supervisor and `pdfs diagnostics` exist to + catch the next one. ## Release assurance | Item | State | |---|---| -| Tag, workspace and PKGBUILD version agreement | Enforced by the release workflow | +| Tag, workspace and PKGBUILD version agree | Enforced by the release workflow | | Clean-install test of the `.deb` and `.rpm` in a VM | Pending (B62) | | Protocol version identity between front ends and daemon | Open (B63) | | SBOM, provenance and checksums for release artifacts | Open (B65). The RPM is GPG-signed | diff --git a/docs/TESTING.md b/docs/TESTING.md index fda9315..8b9036e 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -1,509 +1,265 @@ # Testing -The normal Rust suite exercises the metadata database, sync planner, offline -queue, write staging, cache, and FUSE handler state without requiring a Proton -account: +Three layers check the client, from fast and offline to slow and against a real account: -```console +| Layer | Needs | Runs in CI | +|---|---|---| +| [Unit tests](#unit-tests) | Nothing | Every push and pull request | +| [Simulation runs](#simulation-runs) | `/dev/fuse` | Every push to `main` | +| [Acceptance suite](#acceptance-suite) | Nothing for the reference run; a signed-in account for the rest | The reference run only | + +## Unit tests + +```bash cargo test --workspace cargo clippy --workspace --all-targets -- -D warnings ``` -## Filesystem API and live FUSE acceptance suite +Tests sit in `#[cfg(test)]` modules next to the code. Database tests are in +`crates/pdfs-core/src/db/tests.rs`; a schema change adds one that migrates from the previous +version. -The account-free suite exercises the filesystem syscall contract against a -temporary local directory. This validates the runner and covers creation/open -flags, positioned and vectored I/O, truncation and sparse files, `mmap`, -`sendfile`, `copy_file_range`, durability barriers, namespace/error semantics, -`renameat2` flags, open-file lifetime, names and enumeration, readdir stability -under concurrent mutation, metadata updates, extended attributes, allocation and -punched holes, refusal semantics for unimplemented operations, concurrent I/O, -and real application workloads. It never reads application state or contacts -Proton: +To check the migrations against a real database, point `PDFS_MIGRATE_DB` at it and run the +ignored test. It migrates a copy taken with `VACUUM INTO` and never writes the database named: -```console -scripts/fuse-acceptance.sh --offline-only +```bash +PDFS_MIGRATE_DB=~/.local/state/proton-drive-linux/cache.db \ + cargo test -p pdfs-core a_copy_of_a_real_database -- --ignored ``` -Kernel/FUSE behavior and remote convergence need a real mount. With no -arguments, the script tests the account the running service is logged in to: +## Simulation runs -```console -scripts/fuse-acceptance.sh # every case, every mount kind -scripts/fuse-acceptance.sh --quick # one on-demand/mirror pair instead of four +The simulation runs (`crates/pdfs-fuse/src/sim/`) start real daemons with real FUSE mounts against +an in-memory Drive, `FakeDrive`. The daemon runs as in production (drain workers, sync engine, +event feed, online probe, control socket); only its Drive is fake. `FakeDrive` has the faults the +real one has: listings that lag a create, renames refused for a stale name hash, no atomic +replace, delayed and reordered events, requests that stall or lose their answer, per-request +latency, and blocks of uneven size. + +A run draws a sequence of steps from a seed: syscalls through a client's mount, the link going +down and up, restarts, settles (every queue must drain within a budget), and files deleted while +the drain holds a write to them. Every syscall is checked against an in-memory model of the +filesystem as it happens. At the end every link comes up and the run checks: + +1. **No loss and POSIX.** Drive's copy of each client's folder is what its model says. +2. **No false conflicts.** No conflict copy anywhere; each file has one writer. +3. **Convergence.** Each mount shows Drive's tree. +4. **Liveness.** No syscall took longer than its budget, and every queue drained once the links + were up. A call that never returns is caught by a watchdog that writes every thread's stack to + `stacks.txt` in the run's state directory. + +| Test | Seeds | +|---|---| +| `one_client_on_a_good_link` | 12 | +| `one_client_deleting_files_as_they_upload` | 3 | +| `one_client_with_stale_hashes_and_reordered_echoes` | 6 | +| `one_client_on_a_flaky_link` | 4 | +| `one_client_on_a_slow_link` | 4 | +| `three_clients_one_writer_each` | 3 | + +`sim/daemon.rs` holds scenario tests for single bugs, run the same way. + +The tests mount FUSE, so they are ignored by default: + +```bash +cargo test -p pdfs-fuse --lib sim:: -- --ignored --test-threads=1 # what CI runs +PDFS_SIM_SEEDS=50 cargo test -p pdfs-fuse --lib sim::run -- --ignored --test-threads=1 +PDFS_SIM_SEED=7 RUST_LOG=pdfs_fuse=debug cargo test -p pdfs-fuse --lib \ + sim::run::tests::one_client_on_a_flaky_link -- --ignored --exact --nocapture ``` -An account run does this, in order: +| Variable | Effect | +|---|---| +| `PDFS_SIM_SEEDS` | How many seeds each test runs, from 1 | +| `PDFS_SIM_SEED` | Replay one seed. A failing run prints the seed, the last steps, and where it kept its state | +| `PDFS_SIM_MEASURE` | Run `a_thousand_files_drain_as_fast_on_wifi_as_on_lan`, which times 1,000 files in 100 folders on a LAN and a Wi-Fi profile and prints the times | -1. Runs the account-free contract as the reference. -2. Runs the contract in a fresh `pdfs-acceptance-` folder in My files and - diffs it against the reference. After the last case it reads back a file - the run wrote and records its digest. A read that fails there, as one did - with the link down (B119), is a failed result named "a file of this run - reads back at the end", and the cleanup still runs. -3. Registers two new sync folders under `~/.cache/pdfs-acceptance/run-/` - and runs the contract and the move cases in every on-demand/mirror pairing. -4. Removes everything it created, locally and remotely, and reports a - `cleanup` result. The run fails if anything is left behind. +A failure is a seed that can be replayed, but the daemon uses real threads, so a race may need +several replays. In CI, the `simulation` job uploads the stacks of a hung run as an artifact. -Cleanup identifies remote folders by uid, never by name alone, so nothing of -yours is touched. It removes each sync folder with `sync rm --delete-remote`, -trashes the My files folder, and then deletes both permanently from the trash. -It only counts a folder as gone once a fresh trash listing no longer shows it. -It then reads `pdfs sync queue` and counts any queued op that still names one -of its folders as left behind too, since such an op keeps its staged bytes and -retries forever (B102). +## Acceptance suite -Ops that were queued before a target starts are not the run's. The run prints -one `NOTE` line naming them, and the queue waits ignore them. They are told -apart by id and queue time together, since SQLite can give a new op the id of -one that has drained. +`scripts/fuse-acceptance.sh` checks the filesystem contract a program sees: creation and open +flags, positioned and vectored I/O, truncation and sparse files, `mmap`, `sendfile`, +`copy_file_range`, durability barriers, renames and their `renameat2` flags, open-file lifetime, +names, `readdir` under concurrent change, metadata, extended attributes, holes, refusals of +unsupported operations, concurrent I/O, and real application workloads. -A write is queued when the kernel releases the file, and `close(2)` returns -without waiting for that. A case that reads `pdfs sync queue` right after -writing therefore waits for the ops it expects (`Daemon.queued`) instead of -reading once. "regression B102" saw one revision of two before it did. +> [!CAUTION] +> Every mode except `--offline-only` creates and deletes real files in Proton Drive. Use a test +> account with nothing you cannot lose. -Cleanup also runs on a failure, Ctrl-C, `SIGTERM` or `SIGHUP`. While it runs, -two more interrupts are ignored; a third abandons it. +### Modes -Each run records what it is about to create in -`~/.cache/pdfs-acceptance/run-/manifest.json` before it creates it, and -holds a lock on that directory. If a run dies without cleaning up (`kill -9`, a -hard timeout, a power cut), the next run finds the unlocked manifest and -finishes that cleanup first. It then removes stray `pdfs-acceptance-<32 hex>` -folders that no live run claims: in My files (older than -`PDFS_ACCEPTANCE_REAP_AGE`, default one hour), under the acceptance home, and -in the trash. A run does not start while that reap leaves anything behind. - -The run creates and destroys real remote folders, so a dedicated test account -without irreplaceable data is still the safest target. - -`--live PATH...` tests mounts you set up yourself and leaves the sync folder -configuration alone. It runs destructive POSIX operations only below a fresh -`pdfs-acceptance-*` directory, refuses paths that are not FUSE mounts, and -removes its directory on success, failure, or interruption: - -```console -scripts/fuse-acceptance.sh --live /mnt/on-demand-testmount +```bash +scripts/fuse-acceptance.sh --offline-only # the reference run; no account +scripts/fuse-acceptance.sh # the signed-in account, every mode pairing +scripts/fuse-acceptance.sh --quick # one on-demand/mirror pairing instead of four +scripts/fuse-acceptance.sh --live /mnt/testmount # mounts you set up yourself +scripts/fuse-acceptance.sh --managed-live /mnt/a /mnt/b ``` +**The reference run** always runs first, against a temporary local directory. It records the +facts each case establishes, and every mount is then **compared** against those facts. A case +fails when the mount differs from an ordinary filesystem, not only when it raises. A fact that may +legitimately differ (inode numbers, block counts, which optional syscalls exist) is **noted** +instead, and listed under the target's `accepted` line. The choice is made in the test body, so +the accepted differences are visible where they are accepted. Prefer `record`; use `note` only +for something a network filesystem cannot do. + +**An account run** (no mode): + +1. Runs the reference. +2. Runs the contract in a new `pdfs-acceptance-` folder in My files, and at the end reads back + a file the run wrote. +3. Registers two sync folders under `~/.cache/pdfs-acceptance/run-/` and runs the contract and + the move cases in every pairing of on-demand and mirror. Before a folder switches to on-demand, + a new file is written into it; once mounted, the file must be listed with its bytes. In every + pairing each folder must still hold the file written into it at setup. +4. Removes everything it made, locally and on Drive, and reports `cleanup`. A run that leaves + anything behind fails. + +Cleanup finds remote folders by uid, never by name, so nothing of yours is touched. It removes +each sync folder with `sync rm --delete-remote`, trashes the My files folder, deletes both from +the trash for good, and checks that a fresh trash listing no longer shows them. Queued ops that +still name one of its folders count as left behind. Cleanup also runs on a failure, Ctrl-C, +`SIGTERM` or `SIGHUP`; while it runs, two more interrupts are ignored and a third abandons it. + +Each run writes what it is about to create to `~/.cache/pdfs-acceptance/run-/manifest.json` +first and holds a lock on that directory. A later run finishes the cleanup of a run that died, and +removes stray `pdfs-acceptance-<32 hex>` folders no live run claims: in My files (older than +`PDFS_ACCEPTANCE_REAP_AGE`, default one hour), under the acceptance home, and in the trash. It +does not start while anything is left. + +Ops that were queued before the run started are not the run's: it prints one `NOTE` line naming +them, and its queue waits ignore them. + +**`--live PATH...`** runs the contract on mounts you set up, and leaves the sync configuration +alone. It works only below a new `pdfs-acceptance-*` directory, refuses paths that are not FUSE +mounts, and removes its directory afterwards. The first path must be a FUSE mount; later paths may +be another mount or a mirrored folder. If all paths show the **same remote folder**, set +`PDFS_ACCEPTANCE_CONVERGENCE=1` to also check that bytes written through the first appear through +the others. + +**`--managed-live A B`** takes two existing, empty, unmounted directories, registers them as sync +folders, and runs every pairing as an account run does, then deletes the remote folders. It +refuses non-empty paths, mounts, and paths already in `pdfs sync list`. + ### Reading the output -A run opens with what it is about to test: the mode, the `pdfs` binary and its -version, Python and the kernel, the per-case limit, the options and the start -time. Each target then names its root, the storage behind it, the daemon it -talks to and how many cases it runs, followed by one line per case: +A run opens with the mode, the `pdfs` binary and version, Python, the kernel, the per-case limit +and the options. Each target then names its root, the storage behind it, the daemon and the +number of cases, followed by one line per case: ``` ==> live FUSE /mnt/testmount -> root /mnt/testmount/pdfs-acceptance-… -> storage FUSE mount /mnt/testmount (fuse.protondrive) - -> daemon pdfs 2.8.3, user@proton.me, online, 0 queued ops + -> daemon pdfs 3.0.0, user@proton.me, online, 0 queued ops -> cases 45 of 45 [ 1/45] creation and open flags .................................. ok 0.41s … [21/45] throughput floors ........................................ ok 48.20s write 31 MiB/s, read 54 MiB/s, 4 KiB writes 12.0 MiB/s, metadata 9 ops/s … - [43/45] regression B115: pdfs ls reports a file's real size ...... ok 6.12s -> reference matches the ordinary filesystem in 412 compared observations -> result 44 passed, 1 skipped in 11m02s ``` -A case ends as `ok`, `FAIL`, `TIMEOUT`, `skip` when it does not apply to the -target (no daemon, no share at that role), or `known` when an `open B` case -reproduced a bug that is still open (see below). The dimmed lines under a case -are its details: the reason for a skip, measured rates, a hash. A failed -`regression B` case also points to its entry in `docs/BUGS.md`. When a case -prints something itself, such as a service restart, the case line is printed -again once the case ends. +A case ends as `ok`, `FAIL`, `TIMEOUT`, `skip` (it does not apply to the target), or `known` (an +`open B` case reproduced a bug that is still open). Dimmed lines under a case hold its details. +A failed `regression B` case points to its entry in [BUGS.md](BUGS.md). -The run ends with a table of every target's counts and time, the report paths, -the slowest cases, the cases over `--budget`, the known issues that still -reproduce, any errors the daemon logged, and each failure with its target. The -last line is `PASS:` or `FAIL:` and the totals. Colour is used only on a -terminal and never with `NO_COLOR` set. +The run ends with a table of every target, the report paths, the slowest cases, cases over +`--budget`, known issues that still reproduce, errors the daemon logged, and each failure. The +last line is `PASS:` or `FAIL:` with the totals. Colour is used only on a terminal and never with +`NO_COLOR` set. -### Edge cases and performance +### What the cases cover -Besides the syscall contract, the suite covers: +Besides the syscall contract: -- **Block boundaries.** Files of 0 bytes up to two 4 MiB blocks plus 3 bytes, - including every size next to a boundary, then overwrites that grow, shrink - and reshape them. -- **Unusual names.** A 255-byte name, NFC and NFD forms of the same text, - newline, tab, emoji, characters Windows forbids, leading and trailing - spaces, and a case-only rename. -- **A deep tree and a wide directory.** 24 levels of nesting renamed at the - top; 128 entries created concurrently, concurrent unlinks, then `rm -rf`. - Online, each create is a round trip to the server, and the kernel runs - creates in one directory one at a time. So the wide case gets twice the - per-case timeout and notes the time per create as `wide.create_ms`. -- **Rename patterns.** Swaps, chains, log rotation, replacing a file under an - open reader, the `EISDIR`/`ENOTDIR`/`ENOENT` refusals, and a folder moved out - and back. -- **Handle coherency.** Unsynced writes seen by a second handle, two `O_APPEND` - writers, 30 rewrite generations, and a truncate under a reader. -- **Throughput.** 64 MiB written and read sequentially, 1 MiB in 4 KiB writes, - and metadata operations on 100 files. The rates are printed. The case fails - below `PDFS_ACCEPTANCE_MIN_MIBPS` (default 10) or `PDFS_ACCEPTANCE_MIN_OPS` - (default 20 operations per second, or 3 on a FUSE mount, where each create - and unlink waits for Drive before it returns). +- **Block boundaries.** Files from 0 bytes to two 4 MiB blocks plus 3 bytes, every size next to a + boundary, and overwrites that grow, shrink and reshape them. +- **Unusual names.** 255 bytes, NFC and NFD forms of the same text, newline, tab, emoji, + characters Windows forbids, leading and trailing spaces, a case-only rename. +- **A deep tree and a wide folder.** 24 levels renamed at the top; 128 entries created + concurrently, unlinked concurrently, then `rm -rf`. +- **Rename patterns.** Swaps, chains, log rotation, replacing a file under an open reader, the + `EISDIR`, `ENOTDIR` and `ENOENT` refusals. +- **Handle coherency.** Unsaved writes seen by a second handle, two `O_APPEND` writers, 30 rewrite + generations, a truncate under a reader. +- **Throughput.** 64 MiB written and read, 1 MiB in 4 KiB writes, metadata on 100 files. The case + fails below `PDFS_ACCEPTANCE_MIN_MIBPS` (default 10) or `PDFS_ACCEPTANCE_MIN_OPS` (default 20 + operations per second, 3 on a FUSE mount). `0` turns a floor off. +- **Operations the filesystem does not implement.** `git`, `rsync`, `cp -a`, `tar` and installers + probe for symlinks, hard links, FIFOs, device nodes, locks, `O_DIRECT` and `SEEK_HOLE`. Each must + work or be refused cleanly: a known errno, never `EIO`, never a hang. +- **Moves between locations** (`managed` in `--list`). Trees moved between two sync folders in + both directions, several sources in one `pdfs move`, and a tree through My files and out again. + The refusals must move nothing and lose nothing: a name already taken, a folder into itself, a + mirrored source holding something Drive lacks, a file still open for writing. +- **The CLI.** `pdfs` into a closed pipe exits 0 with nothing on stderr; `pdfs mkdir`, `rename` + and `rm` by local path, and their refusals. -### The mount is diffed against an ordinary filesystem +### Regressions and open bugs -The local reference run is not only a self-test of the runner. Every case -records the facts it established, and a live run is compared against those -recordings, so a case fails when the mount *differs* from an ordinary -filesystem — not only when it raises. The target's `reference` line names the -exact key: +A case named `regression B` reproduces an entry of [BUGS.md](BUGS.md) so it stays fixed. Most +need a live mount and a reachable daemon, and are skipped without one. `--list` prints every case +and the targets it runs on. -``` - -> reference 1 observation of 412 differs from the ordinary filesystem: - ! creation and open flags.truncated.size: live FUSE /mnt/testmount=6 reference=0 -``` +A regression case checks the outcome **on Drive**, so it waits for the queue first. Reading back +through the mount proves nothing: the local caches return the bytes just written whether or not +they reached Drive. A write is queued when the kernel releases the file, after `close(2)` has +returned, so a case that reads `pdfs sync queue` right after writing waits for the ops it expects. -Facts are recorded in one of two ways, chosen in the test body: +A case named `open B` checks a bug that is still open. While it reproduces, the case ends as +`known`: listed at the end, but not a failure. When it passes, the bug is fixed: rename the case to +`regression B` and make it fail with `check`. No case is open at the moment. -- **compared** — a claim about filesystem semantics. The mount must agree. -- **noted** — context that legitimately differs: inode numbers, block counts, - reported mode bits, and which optional syscalls exist at all. Listed under - the target's `accepted` line, never a failure. +Some cases need what a test account cannot provide alone and skip without it: B34 needs a folder +shared read-only by a **second account**, B34b one shared with editor rights, and B79 an on-demand +synced folder. -Because the choice is made at each call site, the set of accepted divergences -is visible in the test that accepts it. There is no separate allowlist to drift -out of date. When adding a case, prefer `record`; reach for `note` only when an -ordinary filesystem can do something a network filesystem genuinely cannot. +### Timeouts, reports and hung mounts -### Operations the filesystem does not implement +A hung FUSE call cannot be interrupted from Python. Each case runs under a soft alarm (`--timeout`, +default 180 s, or `PDFS_ACCEPTANCE_TIMEOUT`) that reports the case and lets cleanup run, backed by +a hard stop that dumps every thread's stack and exits. After a timeout the run checks whether the +mount still answers a `stat` and a listing within 15 s. If it does, the target goes on; if not, its +remaining cases are skipped. Cases that make many round trips by design get twice the limit. -`git`, `rsync`, `cp -a`, `tar`, `df` and most installers probe for symlinks, -hard links, FIFOs, device nodes, `statfs`, file locks, `copy_file_range`, -`O_DIRECT` and `SEEK_HOLE` whether or not a filesystem offers them. The suite -asserts that each either works or is *refused cleanly* — a recognised errno, -never `EIO` and never a hang. A refusal that is merely unimplemented is fine; -one that is dirty breaks software that never asked for the feature. - -### Regressions from the bug ledger - -Cases named `regression B` reproduce a bug from `docs/BUGS.md` so it stays -fixed. B69 (identical rewrite must not fork a `(sync-conflict …)` copy), B70 (an -in-flight `.crdownload` must not be sealed as a revision) and B74 (a file -renamed after close must keep its content) need a real mount and a reachable -daemon; they are skipped, not failed, without one. - -**B100** (a copied tree reads back at its exact sizes) copies 48 files of -assorted sizes into the mount at once, as `cp -r` does. On a live mount it -waits for the queue, checks every size and byte, runs `pdfs refresh` on each -folder to drop the cached listing, and checks again. The refresh is what -brought back the provisional sizes the bug reported. - -**B101** (a write during its create's upload reaches Drive) holds new files open for -different times before writing, so their empty creates land before, during and after the -bytes arrive, then checks every file after the queue drains. - -**B113** (moving a file before its upload makes no conflict) pauses sync with -`pdfs sync pause --for 10m` and writes three files. While their uploads are held, -it renames one, moves one into a folder, and renames and moves the third. It then -resumes sync, waits for the queue, and checks the bytes and names and that no -conflict copy appeared. It skips when sync was already paused. If the run dies -before the resume, the pause ends by itself after ten minutes. - -**B88** (`pdfs mkdir`, `rename` and `rm` by local path) creates, renames and -removes a folder and a file by their paths in the mount. Then it checks the -refusals: `pdfs rename` of the mount's root to its own name, and `pdfs rm` of a -file outside every mount. `pdfs rm` of the root itself is tried only on a sync -folder the run created. In a mirrored sync folder all three commands must be -refused instead, and nothing may change. - -**pdfs output into a closed pipe** runs `pdfs ls`, `pdfs --json ls` and -`pdfs status` with stdout on a pipe whose reader is already gone. Each must exit -0 with nothing on stderr, as `pdfs ls | head -1` needs. - -**B102** (removing a folder drops the uploads queued inside it) uploads a file -into each of two nested folders, pauses sync and rewrites both. While the -revisions are held, it removes one folder with `pdfs rm` and the other through -the mount. Neither revision may stay queued, and neither folder may come back -after the resume. It skips when sync was already paused. - -**B94** (a deleted `.part` file leaves nothing queued) writes a `.part` file, -checks that its create is parked, and unlinks it. The parked op must leave the -queue within 30 seconds. The sweep that lets an hour-old park through is not -driven. - -**B95** (a `.tmp` file uploads at once) writes `index.tmp` and `preview.temp` -and waits for the queue, which fails at once on a parked op. `pdfs ls` must -list both, and their bytes must read back. - -**B111** (renaming a file straight back works) renames a file there and back -eight times through the mount. On a mount it also does so three times with -`pdfs rename`, after the queue is empty, so the renames race nothing but Drive. -It runs against the reference too. - -**B121** (a file renamed and moved at once lands) renames a file, moves it into -a subfolder and renames it back across folders through the mount, with no pause -between the steps. On a mount it then runs `pdfs rename` and `pdfs move` back to -back. The file must end up where it was sent, with its bytes, and the case -reports how long each half took. It runs against the reference too. - -**B98** (an upload superseded by a write ends) writes 64 MiB and waits for its -upload to show in `pdfs transfers`, then appends to the file. The queue must -drain, the old upload must leave the transfer list, and the file must hold -every byte with no conflict copy. It skips when the upload is never seen on -the wire, and it gets twice the per-case timeout. - -**B80** (`pdfs search` finds a new file after a rewrite) writes a file whose -name carries a unique token, waits for the queue, rewrites it and waits again. -`pdfs search` must find it within 30 seconds. - -**B48** (an emptied folder can be removed at once) writes twelve files into a -folder, lists it, and moves one more in from elsewhere. It unlinks every -child, runs `pdfs refresh` on a mount, and checks that nothing is listed and -that `rmdir` works. It runs against the reference too. - -**B114** (a name `pdfs rm` or `rename` took away stops resolving) looks up two -files, removes one with `pdfs rm` and renames the other with `pdfs rename`. At -once, neither old name may be listed or resolve. - -**B115** (`pdfs ls` reports a file's real size) writes files of 6 bytes, 4097 -bytes and one block plus 7, and compares the sizes `pdfs ls` lists with the -written ones, after the upload and again after `pdfs refresh`. - -**B116** (trashing a folder drops the creates queued in its subfolders) makes -`b116/sub/deeper` and waits for Drive. With sync paused, it writes a `.part` -file there and renames it to its final name, so its create is due but held. -It then runs `pdfs rm b116`, and the create must be gone from the queue. It -restores the folder with `pdfs restore` before it resumes sync. If the file -ends up in the mount's root anyway, the case fails and moves the file to the -trash. - -**B117** (a moved file's queued upload keeps its path) rewrites an uploaded -file with sync paused and moves it with `pdfs move`. `pdfs sync queue` must -list the upload under its new path, and the journal must not show the upload -put off with "pending operation deferred". The bytes must arrive at the new -path with no conflict copy. Without `journalctl`, only the queue is checked. - -**B119** (an uploaded file stays cached) writes a small file and one of a block -plus 119 bytes, waits for the queue, and expects `pdfs ls` to list both as -cached, so reading them does not need the network. - -B74 was found by exactly this mechanism: the B70 case failed, and narrowing it -produced a smaller reproduction that got its own case. The defect is fixed and -live-validated (2026-07-26); the case now guards against its return. - -**B79** (an on-demand device folder accepts writes) discovers its target from -`pdfs locations --json` and skips when this machine has no mounted on-demand -folder. It is the cheap standing guard against a *cross-mount* permission -regression: the primary mount holds device-folder nodes too, and the queue guard -intersects what every live inode space says about a uid, so a fail-closed -classification in one mount denies writes in another. - -**B34** (a viewer-role share is read-only) and **B34b** (an editor-role share -still writes) locate their subject through `pdfs shared-with-me --json`, which -now carries `role` and a mount-relative `path`. Both skip when the account has no -accepted share at that role — B34 needs a **second account** to have shared a -folder read-only, which is the one part of the sharing suite that cannot be -self-served. B34b degrades gracefully in the other direction: a shared *file* -gets its mode bits and `W_OK` checked but is never written to, because it is -someone else's document. - -B34's fourth assertion is the load-bearing one: `pending_uploads` and -`pending_changes` must be **unchanged** across the refused writes. Mode bits -alone would pass a fix that still admitted the write into `pending_op`, which is -the actual harm — a background drain failing 403 forever. - -**Synthetic `Shared with me/` directory contract** asserts the virtual directory -enumerates, reports `0555`, and refuses `mkdir`/`rmdir`/`rename`. It skips when -the directory is absent (an account with no accepted shares). - -A regression case asserts against the *remote* outcome, so it must wait for the -queue to drain before reading. Reading straight back through the mount proves -nothing: the local attribute and content caches will happily return the bytes -that were just written even when nothing reached Drive, which is precisely how -B74 stayed invisible. - -### Bugs that are still open - -Cases named `open B` check a bug that `docs/BUGS.md` still lists as open. -While the bug reproduces, the case ends as `known`: it is listed at the end of -the run, but does not fail it. So a known bug does not hide new failures, and -the run notices when a fix lands, because the case then passes. Rename it to -`regression B` then, and fail it with `check` instead of `KnownIssue`. No -bug is open at the moment. B114 to B117 started as such cases. - -### What a run verifies - -A full account run with `--journal-check` checks these `docs/BUGS.md` entries, -many of them fixed but not yet verified: - -- B48, B80, B94, B95, B98, B100, B101, B102, B111, B114, B115, B116, B117, B119 - and B121, each with its own `regression B` case. -- B118, with "application workloads" and "block boundaries", which failed on - it. It needs Drive to be slow to list a new node, so a clean run is weak - evidence. -- B120, with `--journal-check`, but only in a run that goes offline while it - writes. A mkdir made online goes to Drive at once, so only then does a - folder land from the queue. -- B86, with the mode-switch check in the full matrix. -- B104, B106, B108 and B109, with the cases that found them: "rename - patterns", "application workloads" and the account run's own setup. - -B105, B107 and B110 are races. The suite hit each of them once, so a clean run -is weak evidence for them. `--journal-check` reports B110's stall warning, and -every case that writes checks for B105's conflict copies. - -The suite does not drive: - -- the GUI and the photo gallery (B81, B92, B93, B97); -- states it cannot set up: a `parent_uid` cycle (B82), staged bytes that name - no node (B96), a stuck trash refresh (B103), conflict copies left from an - earlier start (B99), a foreign FUSE mount in the local-file index (B90), - revisions without 4 MiB blocks (B85), and several shares from one owner - (B112); -- B94's hour-long expiry of a park; -- load and failure injection (B22, B27, B30, B37, B42). - -Check those by hand, or with the unit tests their entries name. - -### Timeouts, reports, and hung mounts - -A wedged FUSE operation cannot be interrupted from Python, so each case runs -under a soft alarm (`--timeout`, default 180s) that reports the case and lets -cleanup run, backed by a hard stop that dumps every thread's stack and exits. -That backstop is the only way to learn *where* a mount hung. After a timeout the -run checks whether the mount still answers a `stat` and a listing within 15 -seconds. If it does, the target goes on with its next case, so one slow stretch -of the link does not cost every later case. If it does not, the target's later -cases are not run: against a wedged mount they produce noise, not information. -Cases that make many round trips by design ("application workloads", "unusual -but legal names", "a wide directory" and "regression B98") get twice the limit. - -```console -scripts/fuse-acceptance.sh --live /mnt/testmount \ - --timeout 300 --report-junit results.xml --budget 30 -``` - -`--report-json` also writes every recorded observation, which is what to attach -to a bug report. A known issue has the status `known` there; JUnit has no such -status, so `--report-junit` reports it as skipped. `--budget SECONDS` flags cases that pass but got slower — -B5 was a 186 ms-per-call regression that a pass/fail suite could not see. -`--list` prints every case and which targets it runs against, and `--fail-fast` -stops at the first failure. - -`--journal-check` fails the run if `proton-drive.service` logged any error while -it ran. A suite can pass every assertion while the daemon logs failures behind -it; that has happened, and it was caught only by reading the journal by hand. -Two warnings count as errors too: "queued write conflicts; keeping a conflict -copy" (how B113 showed) and "a fuse worker has held the same job for a long -time" (how B110 showed). - -With `--live`, abandoned `pdfs-acceptance-*` roots from a crashed run are -removed at startup, but only if they match the exact generated name and are -over an hour old (`PDFS_ACCEPTANCE_REAP_AGE`), so a concurrent run is never -disturbed. An account run instead uses the manifest and reaper described -above. - -### Automated mode matrix - -An account run drives this matrix itself, in folders it creates. To use -directories of your choice instead, the managed live runner accepts two -existing, empty, unmounted directories. It -registers both as new sync folders, waits for initial synchronization, then -tests all four pairs in order: on-demand/on-demand, on-demand/mirror, -mirror/on-demand, and mirror/mirror. Every transition must become idle, and an -on-demand folder must actually appear as a mount before its contract runs. - -```console -mkdir -p /mnt/pdfs-test-a /mnt/pdfs-test-b -scripts/fuse-acceptance.sh --managed-live /mnt/pdfs-test-a /mnt/pdfs-test-b -``` - -This creates remote folders and permanently deletes those test remotes during -cleanup. It refuses non-empty paths, existing mounts, and paths already present -in `pdfs sync list`. Use `PDFS_ACCEPTANCE_SYNC_TIMEOUT` to extend transition -timeouts and `PDFS_ACCEPTANCE_PDFS` to select a non-installed CLI binary, for -example `target/debug/pdfs`. - -After the contract, each mode pair also runs the move cases (`--list` shows -them as `managed`). They move trees between the two folders in both -directions, move several sources in one `pdfs move`, and pass a tree through -My files and out again. Every case waits for Drive before and after the move. -It then checks that the tree is only at the destination, that every byte -arrived, and that neither side made conflict copies. The refusal cases prove -that nothing moves and nothing is lost when: - -- the destination already has the name; -- a folder would move into itself; -- a mirror source holds something Drive lacks (a symlink); -- a file in an on-demand source is still open for writing. - -The My files case writes a test folder into your real My files and removes it -afterwards. It skips when the daemon reports no My files mount, or when the -locations are on different volumes. - -Before a mirror folder switches to on-demand, the runner writes a new file into -it and syncs. Once the folder is mounted, the file must be listed with its -bytes (B86). Each switch reports a `regression B86` result under a separate -`mode switches` target. The bug only shows in a folder that was mounted in an -earlier pairing, because that mount's listing is stale. That happens in the -third pairing, so `--quick` runs the check but does not replay the bug. - -In every pairing, each folder must still hold the file the runner wrote into it -at setup, byte for byte. These checks report under `mode switches` as well. A -switch to on-demand frees the local copy, so these reads go to Drive. If the -link drops, the daemon fails such a read with EIO once its two-minute read -budget runs out. That is correct, so the runner waits until the daemon reports -online again, tries once more and notes the retry under the result. Only the -second try decides the result. - -### Durability drill - -`--durability` restarts `proton-drive.service` in the middle of the suite and -re-verifies the bytes written before it. `staging/` and `recovery/` can hold the -only copy of a file whose upload has not finished, so this is the case that -proves the queue is persistent rather than in-memory. It is opt-in because it -interrupts a running daemon; `PDFS_ACCEPTANCE_UNIT` selects a different unit. +| Option | Effect | +|---|---| +| `--report-json PATH` | Every result and recorded observation. Attach it to a bug report | +| `--report-junit PATH` | JUnit XML. `known` is reported as skipped | +| `--budget SECONDS` | Flags cases that pass but take longer (`PDFS_ACCEPTANCE_BUDGET`) | +| `--journal-check` | Fails the run if `proton-drive.service` logged an error during it, or the warnings "queued write conflicts; keeping a conflict copy" and "a fuse worker has held the same job for a long time" | +| `--durability` | Restarts the service mid-suite and checks the bytes written before it. Proves the queue survives a restart. `PDFS_ACCEPTANCE_UNIT` names another unit | +| `--fail-fast` | Stops at the first failure | +| `--list` | Prints every case and its targets | ### Selecting cases -During development, `PDFS_ACCEPTANCE_ONLY` runs tests whose descriptive name -contains the supplied text, ignoring case (for example -`PDFS_ACCEPTANCE_ONLY=namespace`). Separate several texts with commas to run -every case that matches any of them (`PDFS_ACCEPTANCE_ONLY=B116,B117`). A -release acceptance run must leave it unset so the complete contract executes. +`PDFS_ACCEPTANCE_ONLY` runs the cases whose name contains the text, ignoring case. Separate several +with commas (`PDFS_ACCEPTANCE_ONLY=B116,B117`). A filter that matches no case at all fails, as the +typo it is. A release run leaves it unset. -A filter naming a live-only case (every `regression B` is one) selects nothing -in the account-free reference run, which is not an error — that target simply has -nothing to do. Only a filter matching *no case at all* fails, as the typo it is. +| Variable | Effect | +|---|---| +| `PDFS_ACCEPTANCE_PDFS` | The `pdfs` binary to use, for example `target/debug/pdfs` | +| `PDFS_ACCEPTANCE_SYNC_TIMEOUT` | Seconds a sync folder may take to settle after a change of mode | +| `PDFS_ACCEPTANCE_REAP_AGE` | Age in seconds after which a stray `pdfs-acceptance-*` folder is removed (default 3600) | +| `PDFS_ACCEPTANCE_CONVERGENCE` | `1` checks that several `--live` paths show the same bytes | +| `PDFS_ACCEPTANCE_MIN_MIBPS`, `PDFS_ACCEPTANCE_MIN_OPS` | Throughput floors | +| `PDFS_ACCEPTANCE_UNIT` | The systemd unit `--durability` restarts | -Passing more paths runs the same filesystem suite independently against each. -The first path must be the FUSE mount under test; later paths may be another -FUSE mount or a normal local mirror filesystem: +## Before a release -```console -PDFS_ACCEPTANCE_SYNC_TIMEOUT=180 \ - scripts/fuse-acceptance.sh --live /mnt/on-demand-testmount /mnt/testmount -``` +1. The quality gates and the simulation runs pass in CI. +2. A full account run with `--journal-check` passes on a test account: + `scripts/fuse-acceptance.sh --journal-check`. +3. Fixes marked *unverified* in [BUGS.md](BUGS.md) that a run now covers are marked verified. -If two paths really are views of the **same remote folder**, enable an -additional byte-for-byte convergence check explicitly: - -```console -PDFS_ACCEPTANCE_CONVERGENCE=1 \ - scripts/fuse-acceptance.sh --live /mnt/first-view /mnt/second-view -``` - -Do not enable that flag merely because two sync folders belong to the same -account. Their configured remote UIDs must be identical. - -Normally the live suite applies the same API contract independently to each -supplied mount. In convergence mode it exercises the primary once and verifies -that the resulting bytes appear through every secondary view. A pass is -strong evidence for those paths, not a blanket data-loss guarantee. Before a -release, also run the recovery drills in `docs/RECOVERY.md` and inspect -`docs/BUGS.md` for live-verification items. - -The runner accepts already-mounted paths instead of starting a daemon. This -keeps authentication and keyring setup explicit and prevents it from silently -reusing or replacing the normal application state directory. +What no automated run covers yet is listed under +[Verification still owed](ROADMAP.md#verification-still-owed). diff --git a/docs/UI_UX_PLAN.md b/docs/UI_UX_PLAN.md deleted file mode 100644 index e93a0e9..0000000 --- a/docs/UI_UX_PLAN.md +++ /dev/null @@ -1,354 +0,0 @@ -# UI/UX audit and improvement plan - -Audit date: 2026-09-23. Scope: `pdfs-gui` (all pages, dialogs, photo viewer), `pdfs-tray`, and the -backend capabilities the UI would need. File references are to `crates/pdfs-gui/src/app/` unless -stated otherwise. - -## 1. Summary - -The client is functionally rich but the UI is built page by page, and it shows: - -- **No sync control surface.** There is no way to pause or resume sync, no list of what is stuck or - failing, and no conflict overview. The backend has no pause at all, no per-op listing, and no - per-file conflict record (conflicts exist only as `(sync-conflict )` files plus activity rows). - The GUI reads `online`, `pending_uploads` and `pending_changes` from `Status` and ignores - `failing_ops`, `failing_error`, `parked_uploads` and `staged_*`. -- **Information architecture is scattered.** Live transfers are a group on *Settings* while the - *Activity* page is history only. *Locations* and *Computers* point at each other. Import lives under - Settings but is launched from Gallery and returns to Settings. Sign-out, quota, mount status, - pinned files, import and developer info all share the Settings page. -- **Chrome is non-standard.** Every page draws its own title and toolbar row under an almost empty - `AdwHeaderBar` whose title is always "Proton Drive". Two stacked bars, wasted height, no current - folder in the title. -- **Icons and wording drift.** Upload uses `document-send-symbolic` (reads as "send"), offline uses - stars (reads as "favorite"), trash icons mean "stop syncing", "remove device" and "remove - bookmark". "Keep offline" / "Unpin" / "Remove offline copy" / "Available offline" are one concept. - "Computers" vs "device", "Gallery" vs "Photos", "mount service" vs "daemon" vs "Connect". -- **Theming is hardcoded.** `PROTON_PURPLE` overrides the system accent; badge, viewer and tile - colours are hex/rgba literals and ignore light, dark and high-contrast modes. -- **Several visible bugs** (section 9). - -## 2. Design principles - -1. **Follow GNOME HIG and current libadwaita.** The system has libadwaita 1.9; the crate is pinned to - feature `v1_5`. Raise it to at least `v1_7` to get `adw::Spinner`, `ToggleGroup`, - `InlineViewSwitcher`, `ButtonRow`, `WrapBox`, `BottomSheet`, and (1.8) `ShortcutsDialog`. -2. **Sync state is first class.** The user must always see: up to date / syncing / paused / offline / - needs attention, and be one click from pausing or from the list of problems. -3. **One term, one icon, one place** per concept (glossary in section 8). -4. **Secondary and destructive actions go into menus.** Rows get at most one visible suffix action - plus a `view-more-symbolic` menu. Destructive actions are red inside the menu and in the confirm - dialog, not as filled buttons in headers. -5. **Never block on the network visibly.** Keep old content on screen while reloading, show a header - spinner, never flash the whole page to a "Loading…" status page. -6. **Respect the system.** Use the system accent by default (brand purple stays on login, app icon - and an opt-in toggle). Use named colours (`@success_color`, `@warning_color`, `@error_color`). - -## 3. New information architecture - -### Sidebar - -``` -Proton Drive (header: app name) - My files - Photos - Shared with me (badge: pending invitations) - Shared by me - Computers - Trash -───────────── - Sync (badge: issue count, icon reflects state) -───────────── footer - [●] Up to date · Pause ← sync status strip, click opens Sync - ▓▓▓░░░ 269 GiB of 1.6 TiB ← quota - (avatar) user@proton.me ⋮ ← account menu: Preferences, Sign out… -``` - -- **Remove** sidebar rows: *Locations* (merged into Sync), *Activity* (becomes the Sync "History" - view), *Settings* (becomes an `AdwPreferencesDialog`, `Ctrl+,`). -- **Rename** *Gallery* to *Photos* and *Shared* to *Shared by me*. -- Primary menu: Preferences, Keyboard Shortcuts, About. The About dialog's `debug_info` gets app - version, user agent, daemon status and config path (replacing the "Developer" group). - -### Sync page (new, replaces Locations + Activity + Settings→Activity) - -Top: an `AdwBanner`-style status card with state icon, one-line summary ("Syncing 12 files · 3.4 -MB/s", "Paused until you resume", "2 files need attention") and one primary button: **Pause** / -**Resume** (split button with "Pause for 1 hour / until tomorrow / until resumed"). - -Below, an `InlineViewSwitcher` with four views: - -1. **Issues** (default when non-empty): conflicts, failing uploads (`last_error`, attempts, next - retry), folders in `error`/`conflict` state, parked uploads older than N minutes. Each row has a - primary fix action (Resolve…, Retry now) and a menu (Show in folder, Discard change…). -2. **Transfers**: in-flight uploads and downloads with progress, speed, cancel; background jobs - (thumbnail build, Takeout import, re-date) with their own cancel. -3. **Folders**: the old Locations list. Row: path, mode as a dropdown suffix (Synced / On-demand, with - confirm when going on-demand deletes local copies), state subtitle, `Sync now` and a menu (Open, - Change location…, Pause this folder, Stop syncing…). "Add folder…" as a `ButtonRow` at the end. - The primary mount appears first and is not removable. -4. **History**: the old Activity feed, with a kind filter, failures in `@error_color`, rows - activatable (open or reveal the target), and "Clear history". - -### Preferences dialog (slimmed Settings) - -| Page | Contents | -|---|---| -| General | Start on login; Mount location (path + Change…); Notifications (on sync issues, on import done) | -| Storage | Cache usage bar; Cache budget (spin row, in GB, "Unlimited" switch instead of magic 0); Clear cache…; Offline files (count + "Manage" → Sync › Folders filter or a sub-page) | -| Sync | Ignore patterns (editable list); Conflict clean-up: Off / Report / Remove identical copies (`conflict_sweep`); later: bandwidth limits | -| Advanced | Open-with rules, search prompt config (only if kept in GUI) | - -Moved out: account row and Sign out (sidebar account menu, with confirm), quota (sidebar footer), -mount status (sidebar status strip), live transfers (Sync), Google Photos import (Photos page menu), -version/user agent (About → Debug info). - -## 4. Backend work required - -Everything below is new; nothing like it exists today. - -### B-1 Pause and resume (global, then per folder) - -- `Request::SetSyncPaused { paused: bool, until: Option }`, persisted (config key - `sync_paused_until` or a DB setting) so a pause survives restarts; `Response::Status` gains - `paused` and `paused_until`. -- Drain: `run_pending_drain` (`pdfs-fuse/src/drain.rs:265`) already releases claimed ops while - offline; gate on `paused` at the same point and reuse the wake path. -- Sync engine: add `SyncMsg::Pause/Resume` or gate `reconcile_all` / `reconcile_folder`; the 120 s - poll and watcher keep recording, they just do not apply. -- FUSE reads must keep working while paused (on-demand hydration is a read, not a sync); document that - pause means "no uploads, no mirror reconcile". -- Per folder later: `sync_folder.paused` column, skipped in `reconcile_all`. -- Bandwidth limits later: token bucket in `CountingReader` / `CountingWriter` - (`pdfs-fuse/src/transfers.rs`). - -### B-2 Pending ops and transfers - -- `Request::ListPendingOps` → id, kind, path/name, attempts, `last_error` (classified as - `ErrorKind`), `next_attempt_at`, parked flag, staged bytes. -- `Request::RetryOp { id }` (set `next_attempt_at = now`, wake drain) and `RetryAllFailing`. -- `Request::DiscardOp { id }` for a create/revision whose bytes should be dropped. Must save the - staged blob to a local recovery folder first; never silent data loss (see `docs/RECOVERY.md`). -- `TransferItem` gains a stable id and `Request::CancelTransfer { id }`. -- Per-folder last error message on `SyncFolderInfo` (today the text only exists in the activity feed). - -### B-3 Conflicts - -- `Request::ListConflicts`: walk nodes whose name parses with `sweep.rs` `conflict_base_name`, for - the mount and every mirror folder. Return copy path, original path (if it exists), both sizes and - mtimes, whether content is identical (size + `content_sha1`, as the sweep already proves), and - which side is local vs remote when known. -- Optionally a `conflict` table written by `keep_as_conflict_copy` / `preserve_conflict_copy` so the - origin (drain vs mirror, device) is recorded instead of inferred from names. -- `Request::ResolveConflict { copy, keep: Original | Copy | Both }`: - - Original: trash the copy (re-verify like the enforcing sweep does). - - Copy: upload the copy's content as a new revision of the original, then trash the copy (the old - content stays in version history, so this is reversible). - - Both: rename the copy to a user-chosen name, drop the `(sync-conflict …)` stamp. -- Activity rows for each resolution. - -## 5. Shell and visual polish (all pages) - -- **Per-page header bars.** Give each stack page its own `adw::ToolbarView` + `adw::HeaderBar` (or - inject page widgets into the shared bar on page switch). Title = page name, or the current folder - in My files. Page actions move into the header; the extra toolbar row disappears. -- **One page skeleton.** Same margins everywhere (today 12 vs 18), same `Clamp` rule (lists that are - settings-like clamp at 720, file/photo grids are full width), same status pages (`compact`, - `vexpand`, crossfade), same Retry semantics (Retry reloads; "Start Proton Drive" is a separate, - explicitly labelled button when the service is down). -- **Loading.** `adw::Spinner` in the header; keep the old model until the new one lands; status page - only on first load. -- **Theme.** Drop the global `accent_*` override; offer "Use Proton purple accent" in Preferences. - Replace hex colours with named colours; test light, dark and high contrast. -- **Move the 70-line CSS string** in `main.rs:232` to `resources/style.css` in the gresource - (`style-dark.css` if needed), so it can be edited and linted. -- **Shortcuts.** Replace the hand-rolled cheatsheet with `adw::ShortcutsDialog` (or - `GtkShortcutsWindow` before 1.8) listing all bindings, including F5/Ctrl+R and the viewer keys. -- **Toasts vs dialogs.** Dialogs report their own errors inline (versions, share, add bookmark); a - toast behind a modal is invisible. Pluralise properly (`ngettext`-ready helpers, no "item(s)"). -- **i18n groundwork.** Wrap user strings in a `gettext!`-style macro now, even without translations, - so later localisation is mechanical. - -## 6. Page-by-page plan - -### My files - -Header: `[‹ ›] Path bar (breadcrumbs, scroll to end, Ctrl+L to edit) … [search] [view ▾] [+ New ▾]` - -- **New ▾** (`list-add-symbolic`, suggested): New folder (Ctrl+Shift+N), Upload files (Ctrl+U), - Upload folder. Replaces three separate icons, fixes the "send" icon. -- **View ▾** split button: Grid/List (Ctrl+1/2), Sort by Name/Size/Modified, ascending/descending, - Folders first, zoom slider. Persist choice (config or GSettings). -- **Folder ⋮ menu** (in header or background context menu): Refresh, Build thumbnails, Make folder - available offline, Open in file manager, Share folder…. Build thumbnails leaves the main toolbar. -- **Navigation.** Back/forward history (Alt+←/→), Up (Alt+↑); searching then pressing Back clears - the search consistently. -- **Context menus** as `gtk::PopoverMenu` from `gio::Menu` + actions: keyboard navigable, Menu / - Shift+F10, same action set as details pane. Add a background menu. Order: Open, Open with…, Play - (media) · Available offline (check item) · Share…, Copy link · Rename…, Move to…, Versions… · - Move to Trash (destructive). -- **Offline state.** One concept "Available offline", toggle semantics. Badges: cloud-only = no - badge or subtle `cloud-outline`; downloaded = check; available offline = pin/`folder-download` - style filled icon. Use named colours. Folders can be made available offline too, or the toggle is - hidden for them (fix `details.rs:259`). -- **List view.** Sortable, resizable columns: Name, Status, Size, Modified (date + time), and - Location for search results. Rubberband selection. One selection model shared between views. -- **Details pane.** On wide windows a real side pane (breakpoint, not overlay); header bar with - close; real thumbnail; sections: Info, Offline, Sharing (people count, link), Versions (latest 3 + - "All versions…"). Same actions as the context menu. -- **Selection bar** (≥1 selected, not ≥2): count + size, Available offline, Move to…, Share (single), - Move to Trash; Select all (Ctrl+A). Drop the confirm dialog for trash since Undo exists (trash is - reversible), or keep confirm and drop Undo — not both. -- **Drag and drop.** Drop external files/folders to upload (reuse Takeout's `FileList` target), drag - the whole selection, highlight drop targets, breadcrumbs accept drops, Undo toast on move. -- **Move to…** a folder-picker dialog (tree/list of Drive folders) instead of a free-text path. -- **Rename** pre-selects the stem, not the extension; drop the redundant "Rename “x”." body. -- **Search.** Scope toggle (This folder / Everywhere), "Showing first 200 results" note, Location - column, correct cached badges (search hits hardcode `cached: false`). -- **Upload feedback.** Toast "Uploading 3 items" with "View" → Sync › Transfers. - -### Photos (was Gallery) - -- Header: title "Photos", `InlineViewSwitcher` Timeline | Albums in the header, Select toggle, - `+ Upload` (fix: use `adw::ButtonContent`, today `.icon_name()` overwrites `.label()` at - `photos.rs:426-432`), ⋮ menu: Import from Google Photos…, Refresh, Rebuild library. -- Filters: a single filter dropdown/`ToggleGroup` (All, Photos, Videos, RAW, Favorites) instead of - two segmented rows where "Photos" appears twice. Counts follow the active filter. -- **Month scrubber** on the right edge that *jumps* (not filters); the month dropdown becomes a real - "Filter by date" if kept. -- Infinite scroll with prefetch (no "Load more" pill; page size ≥ 200; also trigger when the first - page does not fill the viewport). -- Tiles: `ContentFit::Cover` (crop) in justified rows, known aspect ratio from metadata to prevent - reflow, placeholder as a tinted card of the final size, subtle hover (no 1.06 scale with heavy - shadow on a 2 px gap grid), last row justified up to a max stretch. -- Context menu on tiles: Open, Favorite, Add to album, Download/Save copy, Show in My files, Move to - Trash. Selection: Shift range, Ctrl+A, bulk Favorite, Add to album, Download. -- Empty states per filter ("No favorites yet", "No videos in June 2024") instead of the global - "No photos yet" with upload buttons. -- Import: runs show a banner on Photos ("Importing from Google Photos — 1,204 of 5,000 · View") and - Back returns to where the user came from. -- **Albums:** Create, rename, delete, remove from album (not trash). Fix state bugs: filter row - hidden on the album grid, subtitle count, scroll position kept when switching back. -- **Viewer:** `adw::Window` (or full-window overlay), controls auto-hide after 2 s idle, zoom/pan - (pinch, Ctrl+scroll, double-click), info panel beside the image instead of over the top bar, - pages beyond the loaded 60, counter against the real total, videos play inline with - `gtk::Video`, trash with the same confirm/undo rule as the grid, real filename in "Save a copy", - success toast, "Open with…" app chooser, favorite tooltip reflects state, readable dates. - -### Shared with me / Shared by me - -- Shared with me: header title shows the drill-down path, with back; invitations as a banner at - the top ("2 invitations · Review") opening a list, Reject confirmed; bookmarks become a separate - view or section via a switcher. Rows: Open, Save to My files, Available offline where supported. -- Shared by me: rows activatable (open), suffix "Manage" becomes a menu with Copy link, Manage - access…, Stop sharing…. -- Share dialog: inline validation, loading spinner, confirm "Remove access" and "Delete link", - human role names, link expiry. - -### Computers - -- Show the real device name for "This computer" with rename inline. -- Other computers: one suffix menu (Rename, Use this identity…, Remove…). Remove uses a destructive - confirm that requires typing the device name since it deletes backups. -- "Restore folders" becomes a per-device action ("Restore to this computer…") with an inline-empty - state when nothing is restorable, and moves the "Add folder" responsibility fully to Sync. -- Terminology: "computer" everywhere in UI copy. - -### Trash - -- Same skeleton as other list pages (clamped `boxed-list`), sort by deleted date, multi-select with - Restore / Delete permanently, "Empty Trash…" moved into a header ⋮ menu or kept as a flat button - with a destructive confirm, proper plurals, Retry on error. - -### Login - -- Human error messages mapped from error kinds, spinner in the button, "Create account" and "Forgot - password" links, recovery-code hint in 2FA, cancelling CAPTCHA returns to the form with a message. - -## 7. Tray - -- Symbolic icon variants: `pdfs-synced`, `pdfs-syncing`, `pdfs-paused`, `pdfs-offline`, - `pdfs-attention`; tooltip with the one-line state; `NeedsAttention` status for issues. -- Menu: status line · "N issues — View" (opens Sync › Issues) · Pause sync ▸ (1 h, until tomorrow, - until resumed) / Resume · Open Proton Drive · Open folder · "Sign in…" when logged out (no - "Connect") · Stop Proton Drive (confirm) · "Hide tray icon" instead of "Quit" (daemon keeps - running and the label must not suggest otherwise). - -## 8. Glossary (one term, one icon) - -| Concept | UI term | Icon | -|---|---|---| -| Root of Drive | My files | `folder-symbolic` | -| Background service | Proton Drive (running / stopped) — never "daemon" or "mount service" in UI | — | -| Cloud-only file | Online only | none / `cloud-outline` | -| Pinned file | Available offline | pin/download-filled custom icon, not stars | -| Upload | Upload files / Upload folder | `document-upload`-style arrow up, custom pair | -| Folder sync mode | Synced / Online only | — | -| Where My files appears on disk | Proton Drive folder — never "mountpoint" in UI | — | -| Stop syncing a folder | Stop syncing… | `media-playback-stop-symbolic` in a menu, not trash | -| Remove a computer | Remove computer… | destructive, in menu | -| Conflict copy | Conflict | `dialog-warning-symbolic` + warning colour | -| Failure | Failed / Needs attention | `dialog-error-symbolic` + error colour | -| Pause | Pause sync / Resume sync | `media-playback-pause-symbolic` / `-start-` | -| Photos section | Photos | `image-x-generic-symbolic` | -| Sharing | Shared by me / Shared with me | distinct icons (`emblem-shared` / `folder-publicshare`) | - -Spelling: US English throughout ("Favorite", "color"), matching the `pdfs favorite` command and -Proton's own apps. Title Case for -dialog headings and buttons in dialogs, Sentence case for menu items and rows (HIG). - -## 9. Bugs found during the audit (quick wins) - -1. Photos Upload pill loses its label (`photos.rs:426-432`); same for viewer "Show on map" - (`photo_viewer.rs:594-598`). -2. Takeout Back always goes to Settings, sidebar highlights Settings when opened from Photos - (`main.rs:779`, `takeout.rs:131-136`). -3. Thumbnail-build completion message computed then hidden (`browser.rs:1632-1667`). -4. Search results show cached files as cloud-only (`browser.rs:2344`). -5. Details "Available offline" switch shown for folders (`details.rs:259`). -6. Grid and list keep separate selections; details pane can describe a hidden-view item. -7. F2 silently does nothing with multiple items selected. -8. Filter changes on the Albums grid swap to the timeline while the Albums toggle stays active - (`photos.rs:958-997`). -9. Subtitle "60 photos" next to "All 1,416" (loaded count vs total, `photos.rs:2007-2015`); viewer - counter "12 of 60". -10. Viewer info panel covers Close/Trash/Save (`photo_viewer.rs:588` vs `648`). -11. Locations mode switch and removal reload Computers, not Locations; a rejected mode switch stays - flipped until the next tick (`devices.rs:612`, `devices.rs:809`, `locations.rs:374`). -12. Going on-demand deletes local copies without confirmation. -13. Retry on Not-connected silently restarts the systemd unit; Locations' Retry does not. -14. Toasts from Versions/Share dialogs land behind the modal. -15. "Emptied trash —" dangling dash, "item(s)", "archive(s)", raw revision id as version title, - raw lowercase link role. -16. Bulk context menu shows a separator directly under the header when a folder is selected. -17. Sign out, Reject invitation, Remove access, Delete link, tray Disconnect: no confirmation. -18. Stale doc comments (`devices.rs:46-48`, `locations.rs:329-331`, `albums.rs:4`, - `browser.rs:1155`). - -## 10. Phased roadmap - -Each phase is shippable on its own. Estimates are rough, for one developer. - -| Phase | Content | Crates | Size | -|---|---|---|---| -| 0 Quick wins | Section 9 bugs, glossary wording, icon swaps (upload, offline, stop syncing), confirmations, plurals | gui, tray | S (2–3 days) | -| 1 Shell | libadwaita feature bump, per-page header bars, sidebar restructure + footer (status, quota, account), Preferences dialog, About debug info, theme/colour cleanup, CSS to gresource, shortcuts dialog | gui | M (1 week) | -| 2 Sync hub + pause | B-1 global pause, B-2 list/retry pending ops, Sync page (status card, Issues, Transfers, Folders, History), tray states and pause menu, GUI uses `failing_*` / `parked_*` | core, fuse, gui, tray, cli | L (1.5–2 weeks) | -| 3 Conflicts | B-3 list + resolve, Issues rows with a compare/resolve dialog (both versions side by side: size, date, device, preview for images/text), CLI `pdfs conflicts` | core, fuse, gui, cli | M–L | -| 4 File browser | Header layout, New menu, sort/persisted view, PopoverMenu context menus, details side pane, DnD upload, folder picker for Move, search scope | gui | L | -| 5 Photos | Rename, header switcher, filters, infinite scroll + scrubber, tile layout, context menu, album management, viewer rewrite (auto-hide, zoom, inline video) | gui (+ core for album mutations) | L | -| 6 Rest | Shared pages, Computers, Trash, Login, per-folder pause, bandwidth limits, i18n wrapping | all | M | - -Phase 2 is where the user-reported gaps (pause, resume, stuck items) get closed; phase 3 closes the -conflict overview. Phases 0–1 fix most of the "unpolished" impression cheaply and should come first -because later phases build on the new header/page skeleton. - -### Verification per phase - -- `cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, - `cargo test --workspace`. -- Backend phases: DB migration tests in `crates/pdfs-core/src/db/tests.rs` (paused state, conflict - table if added), drain tests proving no op is claimed while paused, resolve-conflict tests proving - the losing side stays recoverable (trash or version history). -- GUI phases: before/after screenshots in `images/` (light and dark), keyboard-only walkthrough of - each page, narrow-window check (sidebar collapse breakpoint).