A switch asked for before any pass saw the folder empty found it still
idle, so the check for what only this computer has ran at once and
refused it for good: every synced path read as "deleted here but not on
Proton Drive yet". The account run hit it in the B255 case.
A local copy gone in full is the wipe guard's case, which the push pass
of a queued switch settles without trashing anything on Drive. The
switch now returns NotNow for it, so the request is queued and that pass
runs first. A partial deletion is still refused as before.
The probe that sees the link return runs 2 s after the drop, then 6 s.
A loaded run could turn the link back on after the first probe, and the
next one came after the test's 3 s were up. The test now checks that the
drain takes the mount offline within 2 s, which is the B189 fix itself,
and counts the 3 s from the mount reporting it is back online.
A mirror folder whose files were all gone from this computer stayed in
error for good. The wipe guard rightly refuses to trash the whole folder
on Drive, but nothing got the folder out of that state: a queued switch
to on-demand waits for a pass that ends idle, and its push pass hit the
same guard. Only removing the folder from sync cleared the baseline that
made the empty folder read as "delete everything".
- The push pass for a switch to on-demand forgets the baseline rows of a
local copy that is gone in full instead of trashing them, so the
switch goes ahead and the mount shows what Drive has.
- New Request::ResetSyncFolder, `pdfs sync reset ID` and "Sync From
Scratch…" in a synced folder's menu clear a mirror folder's baseline
(and a queued switch to on-demand, whose push pass would upload a
duplicate of every file). The next pass compares both sides as on the
first sync: one-sided files are copied across, files that differ keep
both versions, nothing is deleted.
- The wipe guards' messages name the reset; restore logs why it skipped
a folder.
Two simulations cover it. The on-demand one failed in exactly the
reported state (mirror, pending ondemand, error) before the fix.
All 8 catalogs translated.
A copy inside the mount went through the kernel's read/write loop, one
FUSE round trip per page, because copy_file_range answered ENOSYS. The
daemon now copies a range whose source bytes are local (an open write
handle, a queued write, the cache) into the destination's scratch file,
the same way write stages them. A range it would have to download gets
EOPNOTSUPP, which makes the kernel copy that one call itself and keeps
the operation enabled. The copy still uploads: Drive cannot copy into a
file cp has already created.
In the simulation, a warm cp of a 256 MiB file went from about
100 MiB/s to about 450 MiB/s (debug build).
`pdfs copy PATH... NEW_PARENT` sends a new control request, Copy, which
copies a file with its content or a folder with its subtree on Proton
Drive through proton-drive-rs's copy_node. Nothing is downloaded or
uploaded. A name already taken in the destination gets the next free
" (n)" suffix through get_available_name.
A copy cannot wait in the queue, because Drive copies what it holds.
So it is refused offline, and while the mount still owes Drive part of
the node: a local placeholder, a queued op, unsaved writes, or, for a
folder, any queued op at all. Local-first does not change this.
DriveApi gains copy_node and get_available_name, and the fake drive
copies subtrees, so a FUSE sim test covers the copy, the suffix, the
folder subtree and both refusals. The activity feed logs a new Copy
kind, translated in every catalog.
Needs the proton-drive-rs release that adds copy_node; until then, build
with a [patch.crates-io] pointing at proton-sdk-rs.
A mirrored folder assumed it was the only client writing to its folder on
Drive. With a second computer syncing the same folder, edits could be lost:
- The first sync took a same-size file at the same path for the same file.
It now records the baseline only if the local file's SHA-1 matches Drive's
digest, and otherwise keeps a conflict copy (B248).
- An upload replaced a revision another client had committed since the pass
looked. The upload now checks Drive's active revision first, and an edit
that read back with someone else's digest is left for the next pass to keep
as a conflict copy (B249).
- A download recorded a revision committed while it ran as synced, so that
edit never arrived. The baseline now records the revision the pass planned
to download (B250).
- Conflict copies now carry the computer's hostname, so two computers do not
make the same name on Drive (B251).
The baseline gains the revision id it was synced at (schema 44), and a new
revision counts as a remote change even at the same mtime and size.
flush answered ok at once and release synced the scratch file, but close(2)
only returns what flush answers, so a full disk at close was reported to
nobody. flush now syncs a handle with writes on a worker and returns ENOSPC,
EDQUOT or EIO; release then finds the pages on disk instead of writing a
whole file back on the dispatch loop. fsync on a descriptor opened for
reading no longer fails with EBADF (B222).
fallocate passed any mode through and tracked only a punched hole or
growth: a zeroed range came back from the remote base at commit, and a
collapse or insert corrupted the file. Unsupported modes now get EOPNOTSUPP,
zeroed ranges count as written, the real errno is returned, and the state
lock is not held across the syscall (B224).
FUSE delivers `release` after `close()` returns, and the release is what
attaches the written bytes to the queued create. A drain worker that is
free can claim the create first and send the file empty, with the bytes
following as a revision. Tests that hold a create, its reply, or lose
its reply then act on the empty create, and their assertions about the
file's content race the revision. CI saw this as an occasional failure
of a_listing_taken_before_a_create_commits_keeps_the_landed_file.
Add Daemon::write_new, which pauses sync, writes the file, waits until
the pending create carries the bytes, and resumes. Use it for every
write that directly follows a create hold.
A folder pin reads each file's modification time and size before it
downloads the file. If the file's create landed during the download, the
landing set Drive's time, and the pin then stored the blob under the old
time. The file showed as kept offline but read as stale.
After each download, the pin now compares the file's time and size with
the tree's. If they changed, it downloads the file again under the new
values, at most three times.
The simulation gains FakeClient::hold_next_download to force that order.
The account run of 3.3.0 failed "move refuses a file still being
written" on the on-demand to mirror pairing: pdfs move moved the folder
while open.txt in it was open for writing, and the write landed after
the move.
retire_create re-keys a landed create's nodes row to the real uid first
and repoints the tree and the open write handles after. A file opened in
between holds a handle with the placeholder uid. has_pending_work_under
placed each open handle by looking its uid up in the nodes table, which
no longer has the placeholder, so it saw nothing open under the folder,
and with the queue already empty the move went ahead. Open handles are
now placed in the in-memory tree by inode, which a landing create does
not change.
The simulation test holds the drain at that point with the new
FakeClient::hold_next_landing_of_any, and fails without the fix.
A simulated daemon could not add a synced folder, so nothing that moves
between a mount, an on-demand folder and a mirror folder could be tested
without a real account.
Listing and registering devices now go through DriveApi, and the fake
Drive keeps the account's computers, each with a root folder outside My
files. The simulated daemons also read their config.json from a scratch
XDG directory instead of the developer's own: an adopted device of the
real account made every sync folder fail to register against the fake
Drive.
The daemon has kept recursive folder pins for a long time, but the app
offered "Available offline" for files only. The context menu, the
details pane and the selection bar now offer it for folders too, and a
pinned folder carries the offline badge in the browser and the launcher.
A folder pin used to download the whole subtree inside the control
request, so a large folder outlasted the client's 120 s read timeout,
and every pinned file was read into memory before it was cached. The
daemon now records the pin, answers at once and downloads on a
background thread as a job the tray and the app show, stopping when the
folder is unpinned or the daemon stops. Pins of files and folders both
go through fetch_content, which streams to disk and skips fresh blobs.
A folder pin is refused while offline instead of looking kept.
A file kept only because a folder above it is pinned has no pin of its
own, and unpinning it silently did nothing. ListDir now reports
pinned_by_folder; the app shows such an entry as available offline with
the toggle greyed out, and the daemon refuses the unpin with a message
naming the folder. Unpinning a folder no longer evicts files that are
pinned on their own.
Requested on Reddit.
A partial write whose base was not in the content cache queued an
incomplete revision, and every write open over it was refused with EIO
until the drain uploaded it. CI's one-client-lan simulation hit this on
seed 1: a create over a file whose partial edit had not drained yet.
The write open now copies the blob and fills its gaps from the remote
base with Core::fill_gaps, as the drain does, and fails only when that
fill fails (B204).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuyxByBpK3fSWtyPkugeHz
ARCHITECTURE.md described the 2.x write path, where namespace changes
waited for Drive and every mount had its own inode space. Rewrite it
around the local tree (lid, inode = lid + 1, local~<lid> stand-ins), the
queue and its 16 drain workers, sync issues, and the remaining split
between the mount and the mirror engine. Drop the pins.json claim; pins
moved into the database at schema v5.
TESTING.md gains the simulation runs and loses the per-case narrative
that --list and BUGS.md already carry. ROADMAP.md takes over the
post-3.0.0 items from the milestone status file.
Remove MILESTONE-3.0.0-STATUS.md (a working log) and UI_UX_PLAN.md (the
2.x design audit). Code comments and CI that cited the untracked
docs/MILESTONE-3.0.0.md now point at TESTING.md, ARCHITECTURE.md and
ROADMAP.md. Threat-model links follow the section to §10.
The confirmation run on 812eb3d (PDFS_SIM_KNOWN=fail, twelve seeds)
passed every profile, so no open bug is tolerated any more: the call,
hang and damage matchers for B125 to B134 go, and any of them now fails
the seed. B126, B128 and B133 were fixed with B138, B162 and B137;
B134 stays open with local_first off only, which the runs do not mount.
The known-bug confirmation run failed one_client_on_a_flaky_link seeds 6
and 11 on three bugs around a create whose answer was lost:
- B182: after a restart, a write over a file whose create landed but was
never read back forked a conflict copy. A revision of a file that has
no revision of Drive's now adopts the landed node first.
- B183: the SHA-1s a create sent were kept in memory, so after a restart
its trash or retry no longer knew the node Drive made. They are now on
the op's row (pending_op.sent_sha1, schema 42).
- B184: a rename rewrote the op's target, and the trash and the retry
looked for the node under the last name only. Each attempt's parent
and name are recorded (pending_op.sent_to); the trash looks under all
of them, and a retry adopts the node an earlier attempt made and moves
it to the current name.
A refused change was only on the Sync page. A folder listing now carries
each node's issue (by local id, so a file not yet on Drive has it too),
and the browser's badge ranks it first, folders included, with the
issue's explanation as its tooltip. The event sampler publishes Files
when the set of issues changes, so the badge comes and goes with it.
A change Drive will not take could be exported or retried, never let go
of. The Sync page's Discard button (with a confirmation) and
`pdfs sync discard <id>` drop it: a node made here goes, as if deleted
before it was sent; a node Drive has goes back to what Drive has, under
the same inode, with every change queued on it, while what is queued
below it stays. A change on the wire or an open file is refused.
An upload that landed let go of its blob before the tree had its node. A
write open in between took the base it sampled at its start, the revision
the upload replaced, found none of its bytes cached, and queued an
incomplete revision; the next write open failed with EIO.
The drain keeps the landing node aside until the tree has it, and a write
open with no queued revision takes its base from that node, or from the
tree as it is when the handle goes in. The fake Drive can hold the drain
in that window (`hold_next_landing`).
A stop that came while a mirror pass was uploading let the pass go on to
the next level of folders until the stop gave up on it and the runtime's
end cancelled the uploads under it. Once the daemon is stopping, a batch
starts none of its waiting ops, lets the ones sending finish, and ends
the pass; the remote walk and a new pass stop too.
A delete of a file whose create was on the wire withdraws the create into
a trash. When its folder was removed too and that trash landed first,
Drive refused the create, and the drain re-homed the deleted file in the
root; only its missing bytes stopped it. The withdrawn create's trash
then read the trashed folder back into the cache before finding nothing
to trash.
A withdrawn create is no longer re-homed, and its trash is done once the
folder it was sent to is trashed or gone.
The account run's 0-byte read of a new file was B166 too: Drive's
listing had the node from before its create was committed, and the
queue no longer held it when read. The fake Drive can now hold a
folder's listing on its way back with one file listed unrevised.
A listing from Drive was overlaid with the queue as it was after Drive
answered. A create that landed in between was in neither, and a wide
folder on Wi-Fi dropped files that were never unlinked. The queue is now
read before Drive is asked too; a node that left it meanwhile is listed
as the DB has it.
A listing from Drive overlays what is still queued for the folder. A create
Drive has made but whose answer is on its way, or which the drain is still
reading back, was in both: Drive's node under its new uid and the queue's
stand-in, under one name. A node queued in the folder now holds its name there.
A folder removed and made again met the old folder while its trash was on
the wire and adopted it, so the new folder's contents went into the old
one and were trashed with it. The twin search now skips hidden nodes, as
listings do.
The fake Drive can hold a trash (hold_next_trash).
adopt_real_uid asked whether a move was queued only after reading the
node back. A move that landed in between left no op, and the answer still
had the old name, so the mount showed the file under it. Ask first.
The fake Drive can hold a read's answer (hold_answer_to_next_read) so the
test can land the move while the answer is on its way.
Queueing a folder's create left the workers asleep until their 30 s idle
poll; every other enqueue woke them. A file's create still waits for the
release that attaches its bytes.
When the read-back after an upload went unanswered, the drain let go of
the staged blob without caching it. A partial write opened next had
nothing to fill its gaps from and queued an incomplete revision, so the
write open after it was refused with EIO.
The bytes are now cached under the size and mtime the tree still has,
which is the base the next write takes. Their SHA-1 stands in for the
revision id the read-back did not bring, so that write chains onto the
upload and does not fork a conflict copy of itself.
B154 is a regression from the B153 fix: after a failure on a held name the
op woke the ops waiting for it, but the database cannot see a create on the
wire, so the op was due again at once and spent its waits in milliseconds.
No wake while another create on the wire holds the name.
B155: a revision upload that reached Drive but lost its answer left the
file at a revision this daemon never heard of, and the retry kept the write
as a conflict copy of itself. A complete write whose bytes the remote holds,
by size and SHA-1, is sent again instead.
The fake Drive gains hold_reply_to_next_create and
lose_reply_to_next_revision for the two new sim tests.
A retried create now adopts the file its lost answer made when Drive's
SHA-1 matches the staged blob, not only when the file is empty. A delete
after a sent create keeps the op as a trash that finds and trashes the
node that create may have made; a new create of that name waits for it.
The simulation's settle now reads the log, so known damage is explained.
With changes through the mount queued, `pdfs rename` and `pdfs move` went
straight to Drive ahead of a rename the mount still had queued, which then
landed and put its name back: the account run's B121 case on every on-demand
mount. They are now queued whenever the mount would queue, or the node has
anything queued, and rewrite a still-queued create's target.
The B113 acceptance case no longer requires creates to reach Drive at once;
under local-first they are queued and the moves rewrite them.
one_client_on_a_slow_link runs the Wi-Fi faults without outages and has
to drain within each settle. a_thousand_files_drain_as_fast_on_wifi_as_on_lan
times 100 folders and 1,000 files on LAN and Wi-Fi; it only prints for
now and runs when PDFS_SIM_MEASURE is set.
A new folder, a new file, a rename or a delete through the mount no longer
waits for Drive: it is recorded locally, the call returns at once, and the
drain sends it as it does offline. `"local_first": false` in config.json
brings back the old path for this release.
Fixes found by the simulation runs on the way:
- B129: a file deleted while its first upload is on the wire is trashed once
the upload lands.
- B132, B140: files made in a new folder survive a restart; a landed folder
hands its queued children over in the stored node too.
- B138: a create waits for the rename or trash that frees its name.
- B139: a file closed while its first upload lands keeps its last write.
- B141: a rename a worker holds stays queued, and the newer one waits behind
it instead of racing it.
- B142: a rename of a file whose create lands meanwhile goes to the uid Drive
gave it instead of answering EBUSY.
- B143: a create whose read-back fails as the link drops is adopted before
the next op for it is sent, so its next write is not a conflict copy.
Found by the seeded simulation runs (docs/BUGS.md).
- B122: a rename across folders goes to Drive as one move-multiple
request, so a sibling holding the new name in the old folder no longer
fails it with EIO. The drain uses the same request.
- B123: a second write open on a file just written no longer fails with
EIO. An empty file counts as complete, our own renames, moves and
online creates record their echo, a duplicated event is skipped, a
rename keeps the file's mtime while its revision stays, and a landed
upload is cached before open handles are rebased onto it.
- B124: a node put back after a failed replacing rename leaves `hidden`
and its folder is listed again, so it shows in the mount.
- B131: a landed create is read back with the same short retry as an
online create, and its bytes are cached under the real uid before an
open handle is repointed at it.
- B135: an online rename settles the tree through the state guard, so
the database has both folders unlisted before the reply. A lookup
right after it no longer rebuilds the destination without the file.
The simulation runs no longer tolerate B122 or B131.
The runs did not find B105 with its fix reverted. Drive lists a trash
later than the round trips take, so the drain never saw its own
client's delete while it still held the write.
The FakeDrive can now hold the next read of one node on its way to
Drive. A new profile, one-client-deletes, writes a file Drive holds,
waits for the drain to read the node back, deletes the file, and lets
the read go on once Drive shows the trash. A conflict the drain keeps
because a client's own file was trashed now fails the run as a false
conflict. With B105's fix reverted, seeds 1 and 2 fail on it.
Outside the step, such a conflict is counted as B134 when the copy is
named after the file: the drain looked after an online unlink had
trashed the file on Drive but before it dropped the write. Seed 3 of
three-clients came up with it once.
Half the time the step makes the file again before the read goes on.
That is B133: the new file's write takes the op id the delete freed,
and B105's check finds it. The runs count it.
The runs also count B132, an ENOENT after a restart that restored
queued ops, in a seed that lost the link. Seed 4 of one-client-flaky
failed on it. The known-bug matchers now also see what was logged
before the call.
The second half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.2. A run starts one or more real FUSE daemons on a shared
FakeDrive. Each client owns a folder `c<n>` that only it writes. A seed
then draws syscalls through the mounts, link flaps, restarts, pauses and
settles. Every syscall is checked against a model of the client's tree as
it happens. After the last step every link comes up and the run checks
no loss, POSIX behaviour, no false conflicts, convergence and liveness.
- sim::model is the POSIX model: the tree, what each call should answer,
and how it changes the tree.
- sim::run has the profiles: one client on a good link, one on a flaky
link with flaps and restarts, and three clients. PDFS_SIM_SEEDS sets
how many seeds each runs, and PDFS_SIM_SEED replays one.
- sim::watchdog catches a call that never returns. It takes every
thread's stack in the process, since no tracer can stop a thread
blocked on a FUSE request. It then aborts the run's FUSE connections
and writes the stacks next to the run's state.
- The daemons' logs at INFO and above are kept, so a known open bug is
told apart from a new failure. Known bugs are counted, and the seed ends
where the mount leaves the model behind. PDFS_SIM_KNOWN=fail fails on
them instead.
- A restart retries opening the database while the last run still holds
it.
- The FakeDrive restamps a node's modification time on a rename or move,
as Drive does (B113).
The runs mount FUSE, so they are ignored by default. Run them with
`cargo test -p pdfs-fuse --lib sim:: -- --ignored --test-threads=1`.
Test-only; no behaviour change.
nvrl
Claude