A delete of a file whose create was on the wire withdraws the create into
a trash. When its folder was removed too and that trash landed first,
Drive refused the create, and the drain re-homed the deleted file in the
root; only its missing bytes stopped it. The withdrawn create's trash
then read the trashed folder back into the cache before finding nothing
to trash.
A withdrawn create is no longer re-homed, and its trash is done once the
folder it was sent to is trashed or gone.
The hydrate rebuilt each listed folder's listing with a pass over every
entry. With 4,600 listed folders and 37,000 entries that took 1.7 s at
every start before the mount came up; one pass grouping the entries by
parent takes 3 ms.
The account run's 0-byte read of a new file was B166 too: Drive's
listing had the node from before its create was committed, and the
queue no longer held it when read. The fake Drive can now hold a
folder's listing on its way back with one file listed unrevised.
A listing from Drive was overlaid with the queue as it was after Drive
answered. A create that landed in between was in neither, and a wide
folder on Wi-Fi dropped files that were never unlinked. The queue is now
read before Drive is asked too; a node that left it meanwhile is listed
as the DB has it.
A stop joined every background thread for as long as each took. The sync
engine's mirror pass calls Drive with no deadline, and a stop that waited for
it was killed by systemd at TimeoutStopSec. The stop now waits up to 10 s for
the background threads and FUSE workers together, names any still running, and
the runtime shuts down within 2 s.
A listing from Drive overlays what is still queued for the folder. A create
Drive has made but whose answer is on its way, or which the drain is still
reading back, was in both: Drive's node under its new uid and the queue's
stand-in, under one name. A node queued in the folder now holds its name there.
A folder removed and made again met the old folder while its trash was on
the wire and adopted it, so the new folder's contents went into the old
one and were trashed with it. The twin search now skips hidden nodes, as
listings do.
The fake Drive can hold a trash (hold_next_trash).
adopt_real_uid asked whether a move was queued only after reading the
node back. A move that landed in between left no op, and the answer still
had the old name, so the mount showed the file under it. Ask first.
The fake Drive can hold a read's answer (hold_answer_to_next_read) so the
test can land the move while the answer is on its way.
Queueing a folder's create left the workers asleep until their 30 s idle
poll; every other enqueue woke them. A file's create still waits for the
release that attaches its bytes.
A lookup in the root publishes the "Shared with me" entry only if the root
listing did not change while its plan was read, and answered EAGAIN when it
had. Right after a restart the root is listed again as the first lookups come
in, so a write could fail with "Resource temporarily unavailable".
ensure_children now takes the listing again, up to four times.
B141 drops a rename that fails with a newer one queued behind it. The
worker records the failure and releases its claim in two steps, and a
rename queued between them waited behind the old one, which was kept
with a backoff. Released, the old one was not due and the new one was,
so the new one landed first and the old one then put the first name
back.
Releasing the claim now drops the old rename too when a newer one is
queued, and both checks match the node by local id as well as by uid.
When the read-back after an upload went unanswered, the drain let go of
the staged blob without caching it. A partial write opened next had
nothing to fill its gaps from and queued an incomplete revision, so the
write open after it was refused with EIO.
The bytes are now cached under the size and mtime the tree still has,
which is the base the next write takes. Their SHA-1 stands in for the
revision id the read-back did not bring, so that write chains onto the
upload and does not fork a conflict copy of itself.
A file or folder made on the mount used to get a minted `local~<ms>-<n>`
placeholder. It now gets `local~<lid>`, a function of the row it is given
first (`Db::add_local_row`), so the stand-in never changes and always leads
back to the row, and through it to the real uid once the node lands.
`Db::lid_of` reads it without a lookup.
Migration V41 rewrites the placeholders still stored to their row's
stand-in: the row and its node_json, its children, the ops on it or into
it (uid, parent, sidecar uid, a rename's original parent), and its pins.
Step 6a of the plan for 3.0.0 phase 3.
B154 is a regression from the B153 fix: after a failure on a held name the
op woke the ops waiting for it, but the database cannot see a create on the
wire, so the op was due again at once and spent its waits in milliseconds.
No wake while another create on the wire holds the name.
B155: a revision upload that reached Drive but lost its answer left the
file at a revision this daemon never heard of, and the retry kept the write
as a conflict copy of itself. A complete write whose bytes the remote holds,
by size and SHA-1, is sent again instead.
The fake Drive gains hold_reply_to_next_create and
lose_reply_to_next_revision for the two new sim tests.
Adoption asks again under the tree lock whether a rename is queued, and
writes the node row through the lock's outbox, so a rename made while
the create landed is no longer undone in the tree and the row.
An op that finds its name held fails as NameHeld: it backs off by the
minimum without doubling, and wakes the waiters for its name again once
its failure is recorded, so a trash that landed in between is not missed.
A retried create now adopts the file its lost answer made when Drive's
SHA-1 matches the staged blob, not only when the file is empty. A delete
after a sent create keeps the op as a trash that finds and trashes the
node that create may have made; a new create of that name waits for it.
The simulation's settle now reads the log, so known damage is explained.
With changes through the mount queued, `pdfs rename` and `pdfs move` went
straight to Drive ahead of a rename the mount still had queued, which then
landed and put its name back: the account run's B121 case on every on-demand
mount. They are now queued whenever the mount would queue, or the node has
anything queued, and rewrite a still-queued create's target.
The B113 acceptance case no longer requires creates to reach Drive at once;
under local-first they are queued and the moves rewrite them.
Schema 40 adds nodes.parent_lid, backfilled from parent_uid. An upsert
and a new row's stub keep it in step, a folder's new row links the rows
stored before it, and a row merged into a landing folder hands its
children over. A restart places each node under its parent's row as it
is now, so a child whose stored node still names a landed folder's
placeholder lands under the folder's real uid.
Schema 39 adds pending_op.lid and parent_lid, backfilled from nodes. Every
op queued or retargeted gets both, and a create gets its row before it
is queued. The claim reads a parent's uid through parent_lid, and a
landing folder rewrites the ops made inside it by local id instead of by
its placeholder string. Ops queued against a row Drive listed follow it
when it is merged into the landing one.
A folder pinned before its mkdir landed kept the placeholder uid in
pins.uid, which named no node once the landing moved the row to the
real uid. The landing now moves the pin with it.
create_folder, trash_nodes, the twin listing and the uploads waited on
Drive without a limit, so one call that never answered held a drain
thread for good. They now give up after INTERACTIVE_CALL_TIMEOUT, and
uploads after that plus their size at 32 KiB/s; a missed deadline reads
as RequestTimeout, so the op backs off like any lost link.
one_client_on_a_slow_link runs the Wi-Fi faults without outages and has
to drain within each settle. a_thousand_files_drain_as_fast_on_wifi_as_on_lan
times 100 folders and 1,000 files on LAN and Wi-Fi; it only prints for
now and runs when PDFS_SIM_MEASURE is set.
ssh-keyscan sometimes returns no key from the runner, which failed the
AUR publish for 2.8.0 and 2.10.0 at the fingerprint check. Retry the
scan up to five times with a longer timeout before checking the pin.
A change Drive refuses because the storage is full, access was taken away,
its folder is gone, a folder limit is reached or the change was rejected
used to look like any other failure until six attempts had passed. The
drain now sorts the refusal into an issue and stores it on the queued op
(schema 38), so it needs attention at once in the app, tray and CLI. The
change stays queued and is retried; nothing is deleted.
The queue listing carries the issue and whether the op has file content.
ExportPendingOp copies that content to a new file: the Sync page offers it
as an Export button next to each issue, and the CLI as
`pdfs sync export <id> <dest>`. `pdfs sync issues` lists what needs
attention, and `pdfs sync queue` shows the issue per change.
docs/MILESTONE-3.0.0-STATUS.md tracks what is left for 3.0.0.
A file renamed away from a.txt while its create was on the wire, and a new
a.txt made after it, could both land on the first file's node: the second
create found a fresh, empty a.txt and took it for its own unanswered create.
The first file's rename then carried the second one's bytes away.
A create now leaves alone a node that another create is making under the
same name, and one the tree already knows. It waits for the name like for
any other queued change (docs/BUGS.md B147).
- B144: when Drive says a name is taken but nothing local holds it any more,
the rename, move or create is sent once more under the wanted name before
falling back to a conflict name. The trash that freed it could land between
Drive's answer and the check.
- B145: the park sweep no longer drops a parked create whose node row is not
written yet. It waits a minute, and keeps open files, before taking a
missing row for a deleted file. The account run's B70 check failed with
EBUSY on this.
- B146: a rename queued while an upload's read-back is on the wire is kept;
the tree no longer takes back the name Drive still had.
A new folder, a new file, a rename or a delete through the mount no longer
waits for Drive: it is recorded locally, the call returns at once, and the
drain sends it as it does offline. `"local_first": false` in config.json
brings back the old path for this release.
Fixes found by the simulation runs on the way:
- B129: a file deleted while its first upload is on the wire is trashed once
the upload lands.
- B132, B140: files made in a new folder survive a restart; a landed folder
hands its queued children over in the stored node too.
- B138: a create waits for the rename or trash that frees its name.
- B139: a file closed while its first upload lands keeps its last write.
- B141: a rename a worker holds stays queued, and the newer one waits behind
it instead of racing it.
- B142: a rename of a file whose create lands meanwhile goes to the uid Drive
gave it instead of answering EBUSY.
- B143: a create whose read-back fails as the link drops is adopted before
the next op for it is sent, so its next write is not a conflict copy.
A plain rowid is the largest one plus one, so a trash queued in the same
transaction that dropped the newest op took its id. The drain retires the
op it sent by id, so a rename landing deleted the trash queued over it and
the file stayed on Drive (B137). Schema 37 rebuilds pending_op with
AUTOINCREMENT, keeping every id and index.
A dump of a database the 2.8.2 build wrote, with queued ops due, parked,
failing, deferred for access and claimed, and creates queued below a
folder made offline. Its migration checks every column of the queue and
the sealed revisions, as the real-database test now does too.
On-demand sync folders kept their own State, which the drain, the
event applier and the sweeps had to visit one by one (B74). Every mount
now serves from the primary's tree. A session keeps only its root, which
the kernel knows as inode 1, and its notification channel; work that
changes the tree tells every mounted session. Size upgrades become
daemon-wide, and moves compare mounts by root instead of by state.
An ignored test takes a copy of the database named by PDFS_MIGRATE_DB
with VACUUM INTO over a read-only connection, migrates it, and checks
that every queued op survives unchanged, every node keeps its row
number as its lid, and no node is lost or doubled.
A node's inode is now derived from its row's local id instead of a
per-run counter, so the same node keeps the same inode across restarts
and across every place that interns it. A lid handed out before the
row is stored reads as no node.
A rename of a node whose create is still queued rewrites the create's
target, but the drain had already copied the op. When the create
landed, the row was deleted unread and the rename was lost (B136).
finish_create now learns where the create was sent and where it landed,
and queues a rename of the real uid when the row has moved on since.
adopt_real_uid keeps the tree's name and folder while that rename is
queued. A landed folder's children also stop naming its placeholder as
their parent in memory, which put the placeholder back on their rows.
First step of phase 2 of docs/MILESTONE-3.0.0.md. Schema 36 rebuilds
`nodes` with `lid INTEGER PRIMARY KEY AUTOINCREMENT`. Each row takes its
old rowid as its lid, so the search index, keyed by rowid, stays valid.
AUTOINCREMENT keeps a deleted row's lid from coming back. `uid` stays
unique and becomes nullable, for nodes that are not on Drive yet.
A landed create's row now takes the real uid in place, keeping its lid.
When a listing had already stored the real node, its content is kept
under the placeholder's lid, which the node was known by first.
Nothing reads the lid yet apart from `Db::lid_of`.
Found by the seeded simulation runs (docs/BUGS.md).
- B122: a rename across folders goes to Drive as one move-multiple
request, so a sibling holding the new name in the old folder no longer
fails it with EIO. The drain uses the same request.
- B123: a second write open on a file just written no longer fails with
EIO. An empty file counts as complete, our own renames, moves and
online creates record their echo, a duplicated event is skipped, a
rename keeps the file's mtime while its revision stays, and a landed
upload is cached before open handles are rebased onto it.
- B124: a node put back after a failed replacing rename leaves `hidden`
and its folder is listed again, so it shows in the mount.
- B131: a landed create is read back with the same short retry as an
online create, and its bytes are cached under the real uid before an
open handle is repointed at it.
- B135: an online rename settles the tree through the state guard, so
the database has both folders unlisted before the reply. A lookup
right after it no longer rebuilds the destination without the file.
The simulation runs no longer tolerate B122 or B131.
Phase 1 of docs/MILESTONE-3.0.0.md, without a change in behaviour.
- Own sealed revisions live only in the own_sealed_rev table. The
in-memory map in front of it is gone, so revision_conflict gives the
same answer before and after a restart.
- A queued rename records the node's original name next to its original
parent, and keeps both across renames that replace the row. The drain
logs when Drive has the node neither where the rename found it nor
where it takes it.
The SDK's NodeUpdated event carries no revision id, so echoes of our own
changes stay recognised by the self-change timer.
nvrl
Nils Pukropp