mirror of
https://github.com/narrrl/proton-drive-linux.git
synced 2026-11-15 17:41:59 +01:00
fix(fuse,core,gui,cli): resumable, robust Google Photos import (#94)
A Takeout import cut short by a restart could never be resumed: the
duplicate check failed on the draft the interrupted upload left behind
("invalid type: null, expected a string"). SDK 0.12.0 skips such drafts,
lets the import replace them across logins, and removes the draft of a
failed upload (B285, B286).
The import keeps its archives and outcome in the state directory, so a
run the service did not finish is reported as interrupted and can be
resumed from the app or with `pdfs import-google-photos --resume`. An
unreadable archive is left out instead of failing the scan, a failure
keeps its counts and still files albums, and a cancel or shutdown is
only reported when work was left undone (B287).
ImportStatus carries the phase, its progress, the running counts and
what can be resumed; the summary says dry_run, to_upload and
error_kind. The app shows these in the user's language, and --wait
prints progress. An empty gallery waits for a timeline refresh already
running instead of starting a second one.
The scan imports only the Photos service of an export, reads geoData
into the photo's Location, pairs live and motion photos (.MOV, .MP) with
their stills, and dates a photo by name and EXIF before Google's upload
time (B288).
This commit is contained in:
1 parent
2ec3b7ff62
commit
2f2fe96cdf
28 files changed
+5196
-2544
No files matched your search
+125
@@ -109,6 +109,10 @@ a live account since.
|
||||
|
||||
| Entry | Title |
|
||||
|---|---|
|
||||
| [B289](#b289--the-takeout-import-mixes-in-other-services-and-drops-location-and-live-photos) | The Takeout import mixes in other services and drops location and live photos |
|
||||
| [B288](#b288--an-interrupted-takeout-import-reads-as-finished-and-one-error-ends-it) | An interrupted Takeout import reads as finished, and one error ends it |
|
||||
| [B287](#b287--a-photo-draft-left-by-an-interrupted-upload-blocks-that-name) | A photo draft left by an interrupted upload blocks that name |
|
||||
| [B286](#b286--resuming-a-takeout-import-fails-on-a-draft-with-no-content-hash) | Resuming a Takeout import fails on a draft with no content hash |
|
||||
| [B285](#b285--a-restarted-secret-service-leaves-the-process-on-a-dead-session) | A restarted Secret Service leaves the process on a dead session |
|
||||
| [B284](#b284--a-keyring-entry-reported-missing-discards-refreshed-session-tokens) | A keyring entry reported missing discards refreshed session tokens |
|
||||
| [B283](#b283--videos-have-no-thumbnail-outside-photos-and-a-ts-recording-looks-like-source-code) | Videos have no thumbnail outside Photos, and a `.ts` recording looks like source code |
|
||||
@@ -388,6 +392,127 @@ a live account since.
|
||||
|
||||
Newest first.
|
||||
|
||||
### B289 — The Takeout import mixes in other services and drops location and live photos
|
||||
|
||||
**Status:** Fixed, not verified live
|
||||
**Found:** 2026-11-15, auditing the Google Photos import alongside
|
||||
[issue #94](https://github.com/narrrl/proton-drive-linux/issues/94).
|
||||
**Where:** `crates/pdfs-core/src/takeout.rs`; `plan` and `upload_takeout_photo` in
|
||||
`crates/pdfs-fuse/src/takeout.rs`
|
||||
|
||||
**Repro:** Import a Takeout export that also holds `Drive/` or `YouTube/`, with Pixel motion
|
||||
photos, iPhone live photos and photos that have a place in Google Photos.
|
||||
|
||||
**Cause.** Five gaps in what the import carried over:
|
||||
- The service folder was dropped without being checked, so media from other Google services in
|
||||
the same export landed in the photo timeline.
|
||||
- `geoData` was never read, although `docs/PHOTOS.md` said the import takes locations from the
|
||||
export, and the SDK had no way to upload a location.
|
||||
- The two halves of a live or motion photo uploaded as unrelated photos, the `.MOV` half usually
|
||||
without a date, and the `.MP` half of a Pixel motion photo was skipped as unknown.
|
||||
- `creationTime`, the day the photo reached Google, outranked the date in the file name and the
|
||||
photo's own EXIF.
|
||||
- A sidecar with `null` in `favorited`, a numeric `timestamp` or a `0` timestamp lost its other
|
||||
fields or filed the photo under 1970.
|
||||
|
||||
**Fix:** The Photos service is the folder whose sidecars carry `photoTakenTime`, and only it is
|
||||
imported (everything, when no folder does). `geoData`, else `geoDataExif`, becomes the photo's
|
||||
`Location` attribute through SDK 0.12.0's `PhotoUploadMetadata::location`; `0, 0` is no position.
|
||||
Stills and videos with the same stem in one folder (`IMG_1.HEIC` + `IMG_1.MOV`,
|
||||
`PXL_1.MP.jpg` + `PXL_1.MP`) are paired unless their dates are minutes apart: the still is
|
||||
tagged as a live or motion photo, the video is uploaded after it with `main_photo_uid` and takes
|
||||
its date and place. The capture time is `photoTakenTime`, then the name, then EXIF, then
|
||||
`creationTime`. Favorites are also set on photos that were already on the account.
|
||||
|
||||
**Verified:** `takeout::tests` in `pdfs-core` (services, locations, timestamps, live keys) and
|
||||
`pdfs-fuse` (pairing, skew, folded copies). Not run against a real export and account.
|
||||
|
||||
---
|
||||
|
||||
### B288 — An interrupted Takeout import reads as finished, and one error ends it
|
||||
|
||||
**Status:** Fixed, not verified live
|
||||
**Found:** 2026-11-15, auditing the Google Photos import alongside
|
||||
[issue #94](https://github.com/narrrl/proton-drive-linux/issues/94).
|
||||
**Where:** `crates/pdfs-fuse/src/takeout.rs`; `refresh_takeout` in
|
||||
`crates/pdfs-gui/src/app/pages/takeout.rs`; `cmd_import_google_photos` in
|
||||
`crates/pdfs-cli/src/main.rs`
|
||||
|
||||
**Repro:** Start an import of several archives, one of them damaged, and restart the service
|
||||
while it uploads.
|
||||
|
||||
**Cause.**
|
||||
- The import state lived only in memory. After a restart `ImportStatus` had no summary, and the
|
||||
app announced "Import finished". The archives had to be chosen again to continue.
|
||||
- A daemon shutdown broke the upload loop but still filed albums and logged a success.
|
||||
- One unreadable archive failed the scan, and a failed request in the middle of the run
|
||||
discarded every count and skipped the albums for what had uploaded.
|
||||
- The progress total counted photos already on the account, so the bar stopped short of the
|
||||
end; a dry run reported "Would upload: 0"; the job's phases reached the app in English.
|
||||
|
||||
**Fix:** The last run's archives and summary are kept in `takeout-import.json` in the state
|
||||
directory. A run still marked running at start-up is `interrupted`, and `ImportStatus` lists
|
||||
its archives as `resumable`; `ResumeImport` (`pdfs import-google-photos --resume`, **Resume** in
|
||||
the app) starts it again, and the duplicate check skips what is there. An unreadable archive is
|
||||
listed in `unreadable_archives` and the rest imports. A failure keeps the counts and still files
|
||||
albums. `ImportStatus` carries the phase and its own progress, the summary says `dry_run`,
|
||||
`to_upload` and `error_kind`, and the app words them in the user's language. The inline
|
||||
timeline refresh of an empty gallery waits for a refresh already running instead of starting a
|
||||
second one (`refresh_timeline_joined`).
|
||||
|
||||
**Verified:** `takeout::tests` in `pdfs-fuse` (saved state, resumable runs) and `pdfs-core`
|
||||
(unreadable archive), and the CLI tests for `--resume` and the exit status. Not run live.
|
||||
|
||||
---
|
||||
|
||||
### B287 — A photo draft left by an interrupted upload blocks that name
|
||||
|
||||
**Status:** Fixed, not verified live
|
||||
**Found:** 2026-11-15, reading the SDK upload path for
|
||||
[issue #94](https://github.com/narrrl/proton-drive-linux/issues/94).
|
||||
**Where:** `upload_photo_from`, `upload_file_from_inner` and `abandon_file_draft` in
|
||||
`proton-drive-rs` (SDK 0.12.0)
|
||||
|
||||
**Repro:** Interrupt a photo upload after its draft is created, sign in again, and upload a
|
||||
photo with the same name.
|
||||
|
||||
**Cause.** Photo uploads replaced a leftover draft only when its `ClientUID` was the current
|
||||
session's. A new login is a new session, so the next upload of that name failed with 2500. A
|
||||
failed upload also never deleted its own draft, which the C# and TypeScript SDKs do.
|
||||
|
||||
**Fix:** SDK 0.12.0 adds `PhotoUploadMetadata::replace_existing_draft`, which the Takeout import
|
||||
sets, and deletes the draft of a new file or photo whose upload fails. After a failed seal it
|
||||
only does so on a definite rejection, because a seal lost in transit may have been applied.
|
||||
|
||||
**Verified:** SDK unit test `draft_cleanup_only_after_a_definite_seal_rejection`. Not run live.
|
||||
|
||||
---
|
||||
|
||||
### B286 — Resuming a Takeout import fails on a draft with no content hash
|
||||
|
||||
**Status:** Fixed, not verified live
|
||||
**Found:** 2026-11-15, [issue #94](https://github.com/narrrl/proton-drive-linux/issues/94)
|
||||
(3.7.0, Fedora). An import stopped by a restart failed every re-run at once with
|
||||
`check for existing photos: serialization error: invalid type: null, expected a string at line
|
||||
1 column 2063`.
|
||||
**Where:** `FoundPhotoDuplicateDto` in `proton-drive-rs` (SDK 0.11.0)
|
||||
|
||||
**Repro:** Interrupt a photo upload after its draft is created, then run the import again.
|
||||
|
||||
**Cause.** The duplicates endpoint sends `ContentHash` (and can send `Hash`) as `null` for a
|
||||
draft, as its API schema says. The SDK declared both as `String` with `#[serde(default)]`,
|
||||
which covers a missing field but not `null`, so one draft failed the whole batch of 150 names
|
||||
and the import ended before uploading anything.
|
||||
|
||||
**Fix:** SDK 0.12.0 makes both fields optional and skips drafts, trashed links and candidates
|
||||
without hashes, like the TypeScript SDK. Every other defaulted DTO field reads `null` as its
|
||||
default too, and an undecodable response is logged with its path.
|
||||
|
||||
**Verified:** SDK unit tests `duplicate_response_tolerates_null_fields_and_skips_drafts` and
|
||||
`duplicate_response_tolerates_a_null_list`. Not run against the reporter's account.
|
||||
|
||||
---
|
||||
|
||||
### B285 — A restarted Secret Service leaves the process on a dead session
|
||||
|
||||
**Status:** Fixed (unverified)
|
||||
|
||||
+37
-2
@@ -112,14 +112,49 @@ Existing folders stay device backups. Older builds cannot open the upgraded data
|
||||
carry their cover as their thumbnail, so they show it at once on other devices too. A cover
|
||||
image added to a folder later is picked up once the song changes or the service restarts.
|
||||
|
||||
- **Resume a Google Photos import** (B288). The archives and the outcome of the last import are
|
||||
kept, so an import cut short by a restart or a reboot no longer has to be set up again. The
|
||||
Import page offers **Resume**, and `pdfs import-google-photos --resume` does the same. Photos
|
||||
that are already uploaded are skipped as before.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`proton-sdk` and `proton-drive-rs` 0.11.0.** Timestamps are converted where they enter
|
||||
- **`proton-sdk` and `proton-drive-rs` 0.12.0.** Timestamps are converted where they enter
|
||||
from the SDK, and a photo keeps its own modification time now that it no longer follows the
|
||||
capture time. [The SDK review](SDK-0.11.0.md) lists the new features and breaking changes.
|
||||
capture time. 0.12.0 fixes the photo duplicate check and failed-upload drafts below.
|
||||
[The 0.11.0 review](SDK-0.11.0.md) and [the 0.12.0 review](SDK-0.12.0.md) list the new
|
||||
features and breaking changes.
|
||||
|
||||
- **The Google Photos import brings more of the export along** (B289).
|
||||
- Places from Google Photos are kept, so imported photos show up under **Places**.
|
||||
- Live photos and Pixel motion photos stay one photo: the video is filed with its still and
|
||||
takes its date. The `.MP` half of a motion photo is no longer skipped.
|
||||
- Only the Google Photos part of an export is imported. Files from Google Drive, YouTube and
|
||||
other services in the same Takeout stay out of the timeline.
|
||||
- A photo without a date of its own is dated by its file name and then by its EXIF before
|
||||
the day it reached Google Photos.
|
||||
- Favorites are also set on photos that were already in Proton Photos.
|
||||
|
||||
- **The import reports what it is doing** (B288). The Import page shows the current step
|
||||
(reading, checking for photos you already have, uploading, sorting into albums), the counts
|
||||
so far, and a progress bar that reaches the end. A scan says how many photos and how much
|
||||
data an import would upload. `pdfs import-google-photos --wait` prints its progress, and
|
||||
`pdfs import-status` shows the step.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Resuming a Google Photos import no longer fails at once** (#94, B286, B287). A photo upload
|
||||
that was cut short left a draft behind, and every later run stopped with
|
||||
`check for existing photos: serialization error: invalid type: null`. Such drafts are now
|
||||
skipped by the duplicate check and replaced by the next upload of that photo, also after
|
||||
signing in again. A failed upload removes its draft.
|
||||
|
||||
- **One bad archive or error no longer ends the whole import** (B288). An archive that cannot be
|
||||
read is left out and named in the report, and the rest of the export imports. When an import
|
||||
stops early, it keeps the counts of what it did and still files the uploaded photos into their
|
||||
albums. A restart of the service while importing is reported as an interruption, not as a
|
||||
finished import.
|
||||
|
||||
- **Photos zoomed all the way out shows the whole timeline.** The first 200 photos fit on one
|
||||
screen there, and with nothing to scroll no more were loaded. More now load whenever the photos
|
||||
don't fill the window, also after a zoom or a resize.
|
||||
|
||||
@@ -196,6 +196,7 @@ commands are for scripts and for cases where the mount is not available.
|
||||
| `pdfs add-to-album UID PHOTOS...` | Add photos to an album. |
|
||||
| `pdfs remove-from-album UID PHOTOS...` | Take photos out of an album. |
|
||||
| `pdfs import-google-photos ARCHIVES... [--dry-run] [--wait]` | Import a Google Takeout export. Pass every `.zip` of the export. |
|
||||
| `pdfs import-google-photos --resume [--wait]` | Start the last import again with the same archives after it was interrupted, stopped or failed. |
|
||||
| `pdfs import-status` / `pdfs cancel-import` | Follow or stop a running import. |
|
||||
| `pdfs redate-photos [--dry-run] [--from DATE] [--to DATE] [--limit N] [--wait]` | Fix photos filed under their upload day when the file name carries the real date. |
|
||||
| `pdfs redate-status` / `pdfs cancel-redate` | Follow or stop a running re-date. |
|
||||
|
||||
+17
-2
@@ -55,6 +55,7 @@ server learns which areas you look at.
|
||||
2. In the app, open the **Upload** menu on the Photos page and choose **Import from Google
|
||||
Photos…**. Select all the `.zip` files at once, because a photo and its details are often in
|
||||
different parts of the export.
|
||||
3. Choose **Scan** to see what an import would do without uploading anything, then **Import**.
|
||||
|
||||
From a terminal:
|
||||
|
||||
@@ -63,8 +64,22 @@ pdfs import-google-photos --dry-run ~/Downloads/takeout-*.zip # show what woul
|
||||
pdfs import-google-photos --wait ~/Downloads/takeout-*.zip
|
||||
```
|
||||
|
||||
The import recreates your albums and takes capture dates and locations from the export. Photos
|
||||
already in your library are skipped, so an interrupted import can simply be run again.
|
||||
What the import takes from the export:
|
||||
|
||||
- **Albums** are recreated. A photo that is in several albums is uploaded once and added to each.
|
||||
- **Dates** come from the export's details, else from a date in the file name, else from the
|
||||
photo's EXIF.
|
||||
- **Places** come from the export, so imported photos appear under [Places](#places).
|
||||
- **Favorites** are kept.
|
||||
- **Live photos and motion photos** stay one photo: the video is filed with its still.
|
||||
|
||||
What it leaves out: photos in Google's trash, and files from other Google services (Drive,
|
||||
YouTube) that are in the same export. A `.zip` that cannot be read is left out and named in the
|
||||
report; the rest still imports.
|
||||
|
||||
Photos already in your library are skipped, so an interrupted import can simply be run again.
|
||||
If the import was cut short, for example by a restart, the Import page offers **Resume**, and
|
||||
`pdfs import-google-photos --resume` does the same without naming the archives again.
|
||||
`pdfs import-status` shows how far an import is, and `pdfs cancel-import` stops it.
|
||||
|
||||
## Fixing photos filed under the wrong date
|
||||
|
||||
+2
-1
@@ -63,7 +63,8 @@ them along.
|
||||
| Add someone else's photo to your album | Blocked | Needs `copyPhoto` in the SDK |
|
||||
| Photo name search | Not started | Photos search by date and filters only |
|
||||
| Automatic photo upload from a folder | Not started | |
|
||||
| Faster Google Photos import | Not started | Uploads run one at a time. Trash folders of locales missing from `TRASH_FOLDERS` import as photos; motion-photo `.MP` parts are skipped |
|
||||
| Faster Google Photos import | Not started | Uploads run one at a time. Trash folders of locales missing from `TRASH_FOLDERS` import as photos |
|
||||
| More of a Google Photos export | Not started | `.tgz` exports and extracted folders; shared albums told apart from own ones; album descriptions and covers; photo descriptions |
|
||||
| Content-addressed block cache | Idea | Store blocks by hash to deduplicate identical content |
|
||||
|
||||
## Performance and correctness
|
||||
|
||||
Whitespace-only changes.
Reference in new issue
Block a user