fix(fuse,core,gui,cli): resumable, robust Google Photos import (#94)

A Takeout import cut short by a restart could never be resumed: the
duplicate check failed on the draft the interrupted upload left behind
("invalid type: null, expected a string"). SDK 0.12.0 skips such drafts,
lets the import replace them across logins, and removes the draft of a
failed upload (B285, B286).

The import keeps its archives and outcome in the state directory, so a
run the service did not finish is reported as interrupted and can be
resumed from the app or with `pdfs import-google-photos --resume`. An
unreadable archive is left out instead of failing the scan, a failure
keeps its counts and still files albums, and a cancel or shutdown is
only reported when work was left undone (B287).

ImportStatus carries the phase, its progress, the running counts and
what can be resumed; the summary says dry_run, to_upload and
error_kind. The app shows these in the user's language, and --wait
prints progress. An empty gallery waits for a timeline refresh already
running instead of starting a second one.

The scan imports only the Photos service of an export, reads geoData
into the photo's Location, pairs live and motion photos (.MOV, .MP) with
their stills, and dates a photo by name and EXIF before Google's upload
time (B288).
This commit is contained in:
nvrl committed 2026-11-15 17:17:47 +01:00
1 parent 2ec3b7ff62
commit 2f2fe96cdf
28 files changed
+5196 -2544

No files matched your search

+125
View File
@@ -109,6 +109,10 @@ a live account since.
| Entry | Title |
|---|---|
| [B289](#b289--the-takeout-import-mixes-in-other-services-and-drops-location-and-live-photos) | The Takeout import mixes in other services and drops location and live photos |
| [B288](#b288--an-interrupted-takeout-import-reads-as-finished-and-one-error-ends-it) | An interrupted Takeout import reads as finished, and one error ends it |
| [B287](#b287--a-photo-draft-left-by-an-interrupted-upload-blocks-that-name) | A photo draft left by an interrupted upload blocks that name |
| [B286](#b286--resuming-a-takeout-import-fails-on-a-draft-with-no-content-hash) | Resuming a Takeout import fails on a draft with no content hash |
| [B285](#b285--a-restarted-secret-service-leaves-the-process-on-a-dead-session) | A restarted Secret Service leaves the process on a dead session |
| [B284](#b284--a-keyring-entry-reported-missing-discards-refreshed-session-tokens) | A keyring entry reported missing discards refreshed session tokens |
| [B283](#b283--videos-have-no-thumbnail-outside-photos-and-a-ts-recording-looks-like-source-code) | Videos have no thumbnail outside Photos, and a `.ts` recording looks like source code |
@@ -388,6 +392,127 @@ a live account since.
Newest first.
### B289 — The Takeout import mixes in other services and drops location and live photos
**Status:** Fixed, not verified live
**Found:** 2026-11-15, auditing the Google Photos import alongside
[issue #94](https://github.com/narrrl/proton-drive-linux/issues/94).
**Where:** `crates/pdfs-core/src/takeout.rs`; `plan` and `upload_takeout_photo` in
`crates/pdfs-fuse/src/takeout.rs`
**Repro:** Import a Takeout export that also holds `Drive/` or `YouTube/`, with Pixel motion
photos, iPhone live photos and photos that have a place in Google Photos.
**Cause.** Five gaps in what the import carried over:
- The service folder was dropped without being checked, so media from other Google services in
the same export landed in the photo timeline.
- `geoData` was never read, although `docs/PHOTOS.md` said the import takes locations from the
export, and the SDK had no way to upload a location.
- The two halves of a live or motion photo uploaded as unrelated photos, the `.MOV` half usually
without a date, and the `.MP` half of a Pixel motion photo was skipped as unknown.
- `creationTime`, the day the photo reached Google, outranked the date in the file name and the
photo's own EXIF.
- A sidecar with `null` in `favorited`, a numeric `timestamp` or a `0` timestamp lost its other
fields or filed the photo under 1970.
**Fix:** The Photos service is the folder whose sidecars carry `photoTakenTime`, and only it is
imported (everything, when no folder does). `geoData`, else `geoDataExif`, becomes the photo's
`Location` attribute through SDK 0.12.0's `PhotoUploadMetadata::location`; `0, 0` is no position.
Stills and videos with the same stem in one folder (`IMG_1.HEIC` + `IMG_1.MOV`,
`PXL_1.MP.jpg` + `PXL_1.MP`) are paired unless their dates are minutes apart: the still is
tagged as a live or motion photo, the video is uploaded after it with `main_photo_uid` and takes
its date and place. The capture time is `photoTakenTime`, then the name, then EXIF, then
`creationTime`. Favorites are also set on photos that were already on the account.
**Verified:** `takeout::tests` in `pdfs-core` (services, locations, timestamps, live keys) and
`pdfs-fuse` (pairing, skew, folded copies). Not run against a real export and account.
---
### B288 — An interrupted Takeout import reads as finished, and one error ends it
**Status:** Fixed, not verified live
**Found:** 2026-11-15, auditing the Google Photos import alongside
[issue #94](https://github.com/narrrl/proton-drive-linux/issues/94).
**Where:** `crates/pdfs-fuse/src/takeout.rs`; `refresh_takeout` in
`crates/pdfs-gui/src/app/pages/takeout.rs`; `cmd_import_google_photos` in
`crates/pdfs-cli/src/main.rs`
**Repro:** Start an import of several archives, one of them damaged, and restart the service
while it uploads.
**Cause.**
- The import state lived only in memory. After a restart `ImportStatus` had no summary, and the
app announced "Import finished". The archives had to be chosen again to continue.
- A daemon shutdown broke the upload loop but still filed albums and logged a success.
- One unreadable archive failed the scan, and a failed request in the middle of the run
discarded every count and skipped the albums for what had uploaded.
- The progress total counted photos already on the account, so the bar stopped short of the
end; a dry run reported "Would upload: 0"; the job's phases reached the app in English.
**Fix:** The last run's archives and summary are kept in `takeout-import.json` in the state
directory. A run still marked running at start-up is `interrupted`, and `ImportStatus` lists
its archives as `resumable`; `ResumeImport` (`pdfs import-google-photos --resume`, **Resume** in
the app) starts it again, and the duplicate check skips what is there. An unreadable archive is
listed in `unreadable_archives` and the rest imports. A failure keeps the counts and still files
albums. `ImportStatus` carries the phase and its own progress, the summary says `dry_run`,
`to_upload` and `error_kind`, and the app words them in the user's language. The inline
timeline refresh of an empty gallery waits for a refresh already running instead of starting a
second one (`refresh_timeline_joined`).
**Verified:** `takeout::tests` in `pdfs-fuse` (saved state, resumable runs) and `pdfs-core`
(unreadable archive), and the CLI tests for `--resume` and the exit status. Not run live.
---
### B287 — A photo draft left by an interrupted upload blocks that name
**Status:** Fixed, not verified live
**Found:** 2026-11-15, reading the SDK upload path for
[issue #94](https://github.com/narrrl/proton-drive-linux/issues/94).
**Where:** `upload_photo_from`, `upload_file_from_inner` and `abandon_file_draft` in
`proton-drive-rs` (SDK 0.12.0)
**Repro:** Interrupt a photo upload after its draft is created, sign in again, and upload a
photo with the same name.
**Cause.** Photo uploads replaced a leftover draft only when its `ClientUID` was the current
session's. A new login is a new session, so the next upload of that name failed with 2500. A
failed upload also never deleted its own draft, which the C# and TypeScript SDKs do.
**Fix:** SDK 0.12.0 adds `PhotoUploadMetadata::replace_existing_draft`, which the Takeout import
sets, and deletes the draft of a new file or photo whose upload fails. After a failed seal it
only does so on a definite rejection, because a seal lost in transit may have been applied.
**Verified:** SDK unit test `draft_cleanup_only_after_a_definite_seal_rejection`. Not run live.
---
### B286 — Resuming a Takeout import fails on a draft with no content hash
**Status:** Fixed, not verified live
**Found:** 2026-11-15, [issue #94](https://github.com/narrrl/proton-drive-linux/issues/94)
(3.7.0, Fedora). An import stopped by a restart failed every re-run at once with
`check for existing photos: serialization error: invalid type: null, expected a string at line
1 column 2063`.
**Where:** `FoundPhotoDuplicateDto` in `proton-drive-rs` (SDK 0.11.0)
**Repro:** Interrupt a photo upload after its draft is created, then run the import again.
**Cause.** The duplicates endpoint sends `ContentHash` (and can send `Hash`) as `null` for a
draft, as its API schema says. The SDK declared both as `String` with `#[serde(default)]`,
which covers a missing field but not `null`, so one draft failed the whole batch of 150 names
and the import ended before uploading anything.
**Fix:** SDK 0.12.0 makes both fields optional and skips drafts, trashed links and candidates
without hashes, like the TypeScript SDK. Every other defaulted DTO field reads `null` as its
default too, and an undecodable response is logged with its path.
**Verified:** SDK unit tests `duplicate_response_tolerates_null_fields_and_skips_drafts` and
`duplicate_response_tolerates_a_null_list`. Not run against the reporter's account.
---
### B285 — A restarted Secret Service leaves the process on a dead session
**Status:** Fixed (unverified)
+37 -2
View File
@@ -112,14 +112,49 @@ Existing folders stay device backups. Older builds cannot open the upgraded data
carry their cover as their thumbnail, so they show it at once on other devices too. A cover
image added to a folder later is picked up once the song changes or the service restarts.
- **Resume a Google Photos import** (B288). The archives and the outcome of the last import are
kept, so an import cut short by a restart or a reboot no longer has to be set up again. The
Import page offers **Resume**, and `pdfs import-google-photos --resume` does the same. Photos
that are already uploaded are skipped as before.
### Changed
- **`proton-sdk` and `proton-drive-rs` 0.11.0.** Timestamps are converted where they enter
- **`proton-sdk` and `proton-drive-rs` 0.12.0.** Timestamps are converted where they enter
from the SDK, and a photo keeps its own modification time now that it no longer follows the
capture time. [The SDK review](SDK-0.11.0.md) lists the new features and breaking changes.
capture time. 0.12.0 fixes the photo duplicate check and failed-upload drafts below.
[The 0.11.0 review](SDK-0.11.0.md) and [the 0.12.0 review](SDK-0.12.0.md) list the new
features and breaking changes.
- **The Google Photos import brings more of the export along** (B289).
- Places from Google Photos are kept, so imported photos show up under **Places**.
- Live photos and Pixel motion photos stay one photo: the video is filed with its still and
takes its date. The `.MP` half of a motion photo is no longer skipped.
- Only the Google Photos part of an export is imported. Files from Google Drive, YouTube and
other services in the same Takeout stay out of the timeline.
- A photo without a date of its own is dated by its file name and then by its EXIF before
the day it reached Google Photos.
- Favorites are also set on photos that were already in Proton Photos.
- **The import reports what it is doing** (B288). The Import page shows the current step
(reading, checking for photos you already have, uploading, sorting into albums), the counts
so far, and a progress bar that reaches the end. A scan says how many photos and how much
data an import would upload. `pdfs import-google-photos --wait` prints its progress, and
`pdfs import-status` shows the step.
### Fixed
- **Resuming a Google Photos import no longer fails at once** (#94, B286, B287). A photo upload
that was cut short left a draft behind, and every later run stopped with
`check for existing photos: serialization error: invalid type: null`. Such drafts are now
skipped by the duplicate check and replaced by the next upload of that photo, also after
signing in again. A failed upload removes its draft.
- **One bad archive or error no longer ends the whole import** (B288). An archive that cannot be
read is left out and named in the report, and the rest of the export imports. When an import
stops early, it keeps the counts of what it did and still files the uploaded photos into their
albums. A restart of the service while importing is reported as an interruption, not as a
finished import.
- **Photos zoomed all the way out shows the whole timeline.** The first 200 photos fit on one
screen there, and with nothing to scroll no more were loaded. More now load whenever the photos
don't fill the window, also after a zoom or a resize.
+1
View File
@@ -196,6 +196,7 @@ commands are for scripts and for cases where the mount is not available.
| `pdfs add-to-album UID PHOTOS...` | Add photos to an album. |
| `pdfs remove-from-album UID PHOTOS...` | Take photos out of an album. |
| `pdfs import-google-photos ARCHIVES... [--dry-run] [--wait]` | Import a Google Takeout export. Pass every `.zip` of the export. |
| `pdfs import-google-photos --resume [--wait]` | Start the last import again with the same archives after it was interrupted, stopped or failed. |
| `pdfs import-status` / `pdfs cancel-import` | Follow or stop a running import. |
| `pdfs redate-photos [--dry-run] [--from DATE] [--to DATE] [--limit N] [--wait]` | Fix photos filed under their upload day when the file name carries the real date. |
| `pdfs redate-status` / `pdfs cancel-redate` | Follow or stop a running re-date. |
+17 -2
View File
@@ -55,6 +55,7 @@ server learns which areas you look at.
2. In the app, open the **Upload** menu on the Photos page and choose **Import from Google
Photos…**. Select all the `.zip` files at once, because a photo and its details are often in
different parts of the export.
3. Choose **Scan** to see what an import would do without uploading anything, then **Import**.
From a terminal:
@@ -63,8 +64,22 @@ pdfs import-google-photos --dry-run ~/Downloads/takeout-*.zip # show what woul
pdfs import-google-photos --wait ~/Downloads/takeout-*.zip
```
The import recreates your albums and takes capture dates and locations from the export. Photos
already in your library are skipped, so an interrupted import can simply be run again.
What the import takes from the export:
- **Albums** are recreated. A photo that is in several albums is uploaded once and added to each.
- **Dates** come from the export's details, else from a date in the file name, else from the
photo's EXIF.
- **Places** come from the export, so imported photos appear under [Places](#places).
- **Favorites** are kept.
- **Live photos and motion photos** stay one photo: the video is filed with its still.
What it leaves out: photos in Google's trash, and files from other Google services (Drive,
YouTube) that are in the same export. A `.zip` that cannot be read is left out and named in the
report; the rest still imports.
Photos already in your library are skipped, so an interrupted import can simply be run again.
If the import was cut short, for example by a restart, the Import page offers **Resume**, and
`pdfs import-google-photos --resume` does the same without naming the archives again.
`pdfs import-status` shows how far an import is, and `pdfs cancel-import` stops it.
## Fixing photos filed under the wrong date
+2 -1
View File
@@ -63,7 +63,8 @@ them along.
| Add someone else's photo to your album | Blocked | Needs `copyPhoto` in the SDK |
| Photo name search | Not started | Photos search by date and filters only |
| Automatic photo upload from a folder | Not started | |
| Faster Google Photos import | Not started | Uploads run one at a time. Trash folders of locales missing from `TRASH_FOLDERS` import as photos; motion-photo `.MP` parts are skipped |
| Faster Google Photos import | Not started | Uploads run one at a time. Trash folders of locales missing from `TRASH_FOLDERS` import as photos |
| More of a Google Photos export | Not started | `.tgz` exports and extracted folders; shared albums told apart from own ones; album descriptions and covers; photo descriptions |
| Content-addressed block cache | Idea | Store blocks by hash to deduplicate identical content |
## Performance and correctness
View File
Whitespace-only changes.