mirror of
https://github.com/narrrl/proton-drive-linux.git
synced 2026-11-03 16:41:26 +01:00
fix(core,fuse): compact the search indexes and stop pruning after partial scans
FTS5 keeps deleted entries until their segments are merged, which with these indexes is close to never. On one machine local_fts held 317 MiB for 40,000 paths, against 20 MiB rebuilt, and search ran four times slower. The indexes are now compacted once a day after a local scan. A scan that hit the entry cap or lost a batch no longer prunes the paths it never reached, which the next scan used to add back (B217).
This commit is contained in:
1 parent
d6bc921a4b
commit
0f279c7536
5 files changed
+182
-4
No files matched your search
@@ -100,6 +100,7 @@ a live account since.
|
||||
|
||||
| Entry | Title |
|
||||
|---|---|
|
||||
| [B217](#b217--the-search-indexes-grow-to-many-times-their-size-and-slow-search-down) | The search indexes grow to many times their size and slow search down |
|
||||
| [B216](#b216--a-damaged-database-is-rebuilt-without-the-writes-still-queued-in-it) | A damaged database is rebuilt without the writes still queued in it |
|
||||
| [B215](#b215--a-remote-name-holding--or--becomes-a-path-outside-the-sync-folder) | A remote name holding `/` or `..` becomes a path outside the sync folder |
|
||||
| [B214](#b214--deleting-a-mirror-folder-here-trashes-another-devices-new-files-in-it) | Deleting a mirror folder here trashes another device's new files in it |
|
||||
@@ -319,6 +320,33 @@ a live account since.
|
||||
|
||||
Newest first.
|
||||
|
||||
### B217 — The search indexes grow to many times their size and slow search down
|
||||
|
||||
**Status:** Fixed, not verified live
|
||||
**Found:** 2026-10-16, architecture audit (`FINDINGS.md` 2.2.1 and 2.2.3). On a live 567 MiB
|
||||
`cache.db`, `local_fts` held 317 MiB for 40,000 indexed paths. Rebuilt, it was 20 MiB, and a
|
||||
search candidate query went from 27 ms to 7 ms. `nodes_fts` shrank from 21 to 16 MiB.
|
||||
**Where:** `Db::local_finish_scan`, `Db::compact_search_indexes_if_due` in
|
||||
`crates/pdfs-core/src/db/local.rs`; `index_local_files` in `crates/pdfs-fuse/src/background.rs`
|
||||
|
||||
**Repro:** Run the daemon for weeks with a home directory where files come and go, then compare
|
||||
the size of `local_fts_data` with what a `rebuild` leaves.
|
||||
|
||||
**Cause.** FTS5 keeps a deleted entry's data until the segments holding it are merged, and with
|
||||
the few large segments these indexes settle into, that is close to never. The client never ran
|
||||
`optimize` or `rebuild`. The local scan made it worse: a scan that stopped at the entry cap, or
|
||||
lost a batch, still pruned every path it had not reached, and the next scan put them all back.
|
||||
|
||||
**Fix.** The search indexes are compacted at most once a day, at the end of a local scan:
|
||||
`local_fts` is rebuilt from `local_files`, and `nodes_fts` is merged into one segment. Each takes
|
||||
about half a second at that size. A scan that did not see everything closes with
|
||||
`local_finish_partial_scan`, which prunes nothing.
|
||||
|
||||
**Test:** `db::tests::compacting_the_search_indexes_keeps_them_whole_and_runs_once_a_day`,
|
||||
`db::tests::a_partial_local_scan_prunes_nothing`.
|
||||
|
||||
---
|
||||
|
||||
### B216 — A damaged database is rebuilt without the writes still queued in it
|
||||
|
||||
**Status:** Fixed, not verified live
|
||||
|
||||
Reference in new issue
Block a user