mirror of
https://github.com/narrrl/proton-drive-linux.git
synced 2026-11-03 16:41:26 +01:00
When the read-back after an upload went unanswered, the drain let go of the staged blob without caching it. A partial write opened next had nothing to fill its gaps from and queued an incomplete revision, so the write open after it was refused with EIO. The bytes are now cached under the size and mtime the tree still has, which is the base the next write takes. Their SHA-1 stands in for the revision id the read-back did not bring, so that write chains onto the upload and does not fork a conflict copy of itself.
7.6 KiB
7.6 KiB
3.0.0: what is done and what is left
Tracks docs/MILESTONE-3.0.0.md. Update it with every commit that moves a phase.
Done
- Phase 0: fake Drive, simulation runs, CI job (2.9.0).
- Phase 1: queued-op preconditions and own sealed revisions in the database (2.9.0).
- Phase 2: local ids, inode = local id, one tree for every mount (2.10.0).
- Phase 3, first part: mkdir, create, rename, unlink and rmdir on the mount are recorded
locally and sent from the queue;
"local_first": falsebrings back the old path (5e0f30c). - Fixes from the simulation and account runs since then: B127 and B141 to B157.
- Sync issues, first cut (§6.2): refusals sorted into issues, shown at once in the app,
tray and CLI; Export button and
pdfs sync export;pdfs sync issues.
Phase 3: left for 3.0.0
- Sync issues (§6.2): first cut done.
- The drain sorts a refusal from Drive into an issue kind (quota, access, missing, limit, rejected) and stores it on the queued op (schema 38).
- An op with an issue counts as "needs attention" at once, not after six failures.
- The queue listing carries the issue kind and whether its content can be exported.
- Export a queued file's content (control request,
pdfs sync export). pdfs sync issues, and the issue shown inpdfs sync queue.- The app's Sync page shows each issue with what the user can do; Export and Retry buttons; translated in every catalog.
- Discard an issue on purpose (after export, or for a change with no content), and undo the local change it stood for. Not in the first cut.
- Show the issue on the node itself (file browser emblem or status). Not in the first cut.
- Planner and executor (§5.4, §5.5): ops keyed by local id; remove
mint_local_uid,finish_createandadopt_real_uid; a landing create setsremote_uidon its row.- Queued ops carry
lidandparent_lid(schema 39). The claim reads the parent's uid throughparent_lid, and a landing folder rewrites the ops inside it by local id. - Node rows link to their parent by lid (
nodes.parent_lid, schema 40), and a restart places each node under its parent's row as it is now. - Queries over children, subtrees and ancestors by
parent_lid, and no rewrite of the children'sparent_uidwhen a folder lands (R3), nor of the ops made inside it (R2). - No more placeholders: a node made here goes by its local id until it lands. In four
parts:
- 6a: a node made here goes by
local~<lid>, a function of its row, instead of a mintedlocal~<ms>-<n>; schema 41 rewrites the old placeholders (nodes, ops, pins). - 6b: an op is found by its node's local id under either uid, the claim keeps one op
per local id on the wire, and the drain sends the uid the row has now;
remap_local_uidis gone. - 6c: children, subtrees, ancestors and ops' parents by lid, so a landing create
rewrites no other row or op; what is read for a uid names the parent as its row has it
now. The row keeps its stand-in in
nodes.uidrather thanNULL: it is as stable, andNULLwould have changed some 40 node queries for no change in behaviour. Pins stay keyed by uid and move with the row in the landing transaction (B148). - 6d: a node written under its stand-in after it landed updates its landed row
instead of bringing the placeholder row back, so a write that finds its create
landed takes the uid from the row and no longer polls the tree for up to 2 s
(
landed_uidis gone). In memory,pending, the uploads and the tree stay keyed by the uid Drive knows: ops are matched by lid in the database (6b),hiddenandcreatingonly ever hold remote uids, andlanded_row_uidis the one place at the mount's edge where a stand-in a handle still holds becomes the real uid.
- 6a: a node made here goes by
- An executor that runs ops in dependency order, many at once: 16 drain workers instead of 3, and a worker that lands an op wakes the idle ones, which slept through the ops it made claimable. The claim already keeps the order: a create waits for its parent's uid, a node's ops go one at a time. Ops stay blocking calls on worker threads: each one spends its time waiting on Drive, so tasks would buy nothing threads do not.
- Queued ops carry
- Speed (§7): run independent ops in parallel, batch trashes and moves; measure 1,000 files
and 100 folders on Wi-Fi against LAN (target: within 10 %).
- The measurement:
a_thousand_files_drain_as_fast_on_wifi_as_on_laninsim/daemon.rs, run withPDFS_SIM_MEASURE=1. It prints the times and asserts nothing yet. - Baseline on
eb1c105, three drain threads: the syscalls take 3.7 s on LAN and 4.3 s on Wi-Fi; the drain takes 138 s on LAN and 938 s on Wi-Fi, 6.8 times as long. - Sixteen workers: the syscalls take 5.2 s on LAN and 5.0 s on Wi-Fi, which meets the target; the drain takes 47 s on LAN and 247 s on Wi-Fi. Most of a create on Wi-Fi is its read-back waiting for Drive to list the new node (250 ms, then 1 s), so batching (step 8) helps trashes and moves, not creates.
- The measurement:
- Done-when checks:
- Simulation passes on the Wi-Fi profile:
one_client_on_a_slow_linkfailed in CI on seeds 2 and 3. Two bugs (B127, B151) and a harness gap: a settle did not read the log, so it could not explain a known bug's damage. All three fixed; not yet confirmed in CI. Replays then found two races on the LAN profile too (B152, B153), both fixed. CI on7269cb5failed LAN seed 9, a regression from the B153 fix (B154), and Wi-Fi seed 2, a write whose answer was lost (B155). Both fixed. One slow-link run of seed 2 failed once with a rename answeringENOENTas a folder move landed. Not reproduced since; it may be B152. Replays of slow-link seed 4 failed two of six times on a file written again after its create lost its answer (B156, fixed). With sixteen drain workers (16fca10), seed 4 failed every replay on a write after an upload that was not read back (B157, fixed). - Known-bug list in
sim/run.rsis empty: B125 and B130 open; B127, B129 and B132 fixed but unconfirmed. - A clean account run. The one on
a6f36d5(2026-10-04, LAN: 427 passed, 12 failed) failed B113 and B121 on every on-demand mount, two mirror-to-mirror move cases, and the journal check. B121 was a bug (B149, fixed); B113's case still expected creates to reach Drive at once and now accepts queued ones. The move cases were the harness: it retried every source after one was busy, and expected a refusal where a mirror to mirror move carries the local copy along. The journal showed a conflict copy retrying into a deleted folder (B150, fixed). Not confirmed yet.
- Simulation passes on the Wi-Fi profile:
- Release: 2.10.0 is tagged; the 2.9.0 tag still waits for the user's go-ahead.
After 3.0.0
- Remove the old online path in
filesystem.rsand thelocal_firstswitch (the release after 3.0.0). - Phase 4: one applier for remote changes.
- Phase 5: mirror folders on the same planner and executor.
- Any time:
Sizeand block geometry as types; a deadline on the remote calls outside the drain (the drain's have one:link::upload_deadline,Core::block_on_within). - Open questions in §12 of the milestone.
PLAN.md: the acceptance script notices network drops and reportsinterruptedinstead of TIMEOUT.