mirror of
https://github.com/narrrl/proton-stream.git
synced 2026-11-03 16:41:26 +01:00
fix(core,meta,app,android,cli): number episodes the way releases do
AniList's per-season lookup searched for "<title> 2nd Season" and the scorer picked the base entry again, so Attack on Titan seasons 2-4 and Jigokuraku season 2 were captioned with season one's names. Episode names came from streamingEpisodes, which is partial and numbered however a streaming site counted. Bleach S17 and Mushishi S02/S03 had none. AniList now walks SEQUEL relations from the match and takes each entry's episodes from ani.zip, which carries TVDB's season, number and absolute number. An episode row keeps all three plus its place in the AniList entry. EpisodeGuide::get matches whichever numbering a file uses and never answers from another season. Episodes and fanart are stored as one enrichment. The episode list is replaced even when empty, and the enrichment is recorded as asked, so a film is not re-asked every run. Storing a different match drops the old entry's episode rows. AniList's 4.75:1 bannerImage is no longer stored as the 16:9 backdrop; ani.zip's 1920x1080 fanart is used instead. Schema v9 adds the columns and deletes the AniList episode rows and banners. Matches are kept. Desktop, Android and the CLI now share service::plan and MetadataService::work. The settings text names ani.zip. Audit and roadmap in docs/METADATA.md; bug entry B61.
This commit is contained in:
nvrl committed
1 parent
52d27ec996
commit
4c80076762
17 files changed
+1665
-603
No files matched your search
@@ -72,7 +72,7 @@ against the SDK's own range download.
|
||||
| Transcoding | None. mpv demuxes and decodes natively. |
|
||||
| UI | egui/eframe on glow; mpv renders into the same GL context. |
|
||||
| Themes | The shipped near-black palette, Catppuccin Latte, Frappé, Macchiato and Mocha, and a Persona 5 one — black, white and one loud red, which is what the Red accent is there for — each with a choice of nine accents, drawn as a gradient unless you turn that off. Picked on the Settings page, applied without a restart. |
|
||||
| Metadata | Filename parsing, with optional AniList/TMDB enrichment — **off by default**, because enabling it sends your library's titles to a third party. It brings posters, synopses and per-episode titles; a title the scorer refuses to guess about can be pinned by hand, and a pinned match is never overwritten. |
|
||||
| Metadata | Filename parsing, with optional AniList/TMDB enrichment — **off by default**, because enabling it sends your library's titles to a third party. It brings posters, synopses, fanart and per-episode titles and synopses (for AniList, from ani.zip, which is sent the matched ids); a title the scorer refuses to guess about can be pinned by hand, and a pinned match is never overwritten. See [docs/METADATA.md](docs/METADATA.md). |
|
||||
| Platforms | Linux and Windows; macOS packaging is written but has not been run on a Mac. A native Android 12+ client is available featuring a Material 3 UI and immersive fullscreen playback. |
|
||||
|
||||
## Building
|
||||
|
||||
@@ -420,7 +420,7 @@ private fun MetadataSettingsDialog(
|
||||
title = { Text("Metadata enrichment") },
|
||||
text = {
|
||||
Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
Text("Off by default: enabling sends the titles in your library to a third party, associated with your IP address and subject to their privacy policy.")
|
||||
Text("Off by default: enabling sends the titles in your library to a third party, associated with your IP address and subject to their privacy policy. With AniList, the matched ids also go to ani.zip for episode names and wide artwork.")
|
||||
SettingToggle("Enable enrichment", enabled) { enabled = it }
|
||||
Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
|
||||
MetadataProvider.entries.forEach { option ->
|
||||
|
||||
@@ -1420,44 +1420,22 @@ impl AndroidEngine {
|
||||
impl AndroidEngine {
|
||||
/// One enrichment pass over the whole library.
|
||||
///
|
||||
/// Two kinds of work, and the difference is a request saved: a title that
|
||||
/// already has a good match but no episode list needs the episode request
|
||||
/// only. Both run fanned out under a semaphore, and a provider error counts
|
||||
/// against the run rather than ending it — one rate-limited title must not
|
||||
/// leave the other thirty unenriched. This mirrors the desktop engine's
|
||||
/// `run_match`, which is the reference for the behaviour.
|
||||
/// What to do per title is decided by `pstr_meta::service::plan`, the same
|
||||
/// as on the desktop. The work runs fanned out under a semaphore, and a
|
||||
/// provider error counts against the run rather than ending it — one
|
||||
/// rate-limited title must not leave the other thirty unenriched. This
|
||||
/// mirrors the desktop engine's `run_match`, which is the reference for the
|
||||
/// behaviour.
|
||||
async fn match_titles_inner(&self, force: bool) -> Result<MatchSummary, BridgeError> {
|
||||
let service = Arc::new(self.metadata_service()?);
|
||||
let provider = service.provider();
|
||||
let (titles, stored) = self.titles_and_metadata()?;
|
||||
let listed = self
|
||||
let enriched = self
|
||||
.catalog
|
||||
.lock()
|
||||
.episode_metadata_ages()
|
||||
.enrichment_ages()
|
||||
.map_err(BridgeError::from_display)?;
|
||||
|
||||
let mut pending: Vec<Work> = Vec::new();
|
||||
for title in titles {
|
||||
let record = stored.get(&title.key);
|
||||
// A hand-picked entry is never searched for again, not even by
|
||||
// "match again" — that button means "the automatic answers are
|
||||
// wrong", and re-deciding the one title the viewer already fixed by
|
||||
// hand is the opposite of what they asked for. Its episode list is
|
||||
// still fetched below if it is missing.
|
||||
let pinned = record.is_some_and(|record| record.manual && record.provider == provider);
|
||||
if !pinned && (force || !pstr_meta::service::is_usable(record, provider)) {
|
||||
pending.push(Work::Match(title));
|
||||
continue;
|
||||
}
|
||||
let has_episodes = listed
|
||||
.get(&title.key)
|
||||
.is_some_and(|(asked, _)| *asked == provider);
|
||||
if let Some(found) = record.and_then(|record| record.metadata.clone())
|
||||
&& !has_episodes
|
||||
{
|
||||
pending.push(Work::Episodes(title, Box::new(found)));
|
||||
}
|
||||
}
|
||||
let pending = pstr_meta::service::plan(titles, &stored, &enriched, provider, force);
|
||||
|
||||
let permits = Arc::new(tokio::sync::Semaphore::new(LOOKUP_CONCURRENCY));
|
||||
let mut lookups = tokio::task::JoinSet::new();
|
||||
@@ -1466,32 +1444,15 @@ impl AndroidEngine {
|
||||
let permits = Arc::clone(&permits);
|
||||
lookups.spawn(async move {
|
||||
let _permit = permits.acquire().await.ok()?;
|
||||
Some(match work {
|
||||
Work::Match(title) => match service.record(&title).await {
|
||||
Ok(record) => {
|
||||
// Only for a title that matched: there is no id to
|
||||
// ask about otherwise.
|
||||
let episodes = match &record.metadata {
|
||||
Some(found) => service.title_episodes(&title, found).await,
|
||||
None => Vec::new(),
|
||||
};
|
||||
Ok((record.title_key.clone(), Some(record), episodes))
|
||||
}
|
||||
Err(error) => Err(error.to_string()),
|
||||
},
|
||||
Work::Episodes(title, found) => {
|
||||
let episodes = service.title_episodes(&title, &found).await;
|
||||
Ok((title.key, None, episodes))
|
||||
}
|
||||
})
|
||||
Some(service.work(&work).await.map_err(|error| error.to_string()))
|
||||
});
|
||||
}
|
||||
|
||||
let mut summary = MatchSummary::default();
|
||||
while let Some(joined) = lookups.join_next().await {
|
||||
match joined.map_err(BridgeError::from_display)? {
|
||||
Some(Ok((title_key, record, episodes))) => {
|
||||
if let Some(record) = &record {
|
||||
Some(Ok(outcome)) => {
|
||||
if let Some(record) = &outcome.record {
|
||||
if record.metadata.is_some() {
|
||||
summary.matched += 1;
|
||||
} else {
|
||||
@@ -1502,16 +1463,16 @@ impl AndroidEngine {
|
||||
log::warn!("store metadata for {}: {error}", record.title_key);
|
||||
}
|
||||
}
|
||||
if !episodes.is_empty() {
|
||||
summary.episodes += episodes.len() as u32;
|
||||
let stored = self.catalog.lock().set_episode_metadata(
|
||||
&title_key,
|
||||
if let Some(enrichment) = &outcome.enrichment {
|
||||
summary.episodes += enrichment.episodes.len() as u32;
|
||||
let stored = self.catalog.lock().set_enrichment(
|
||||
&outcome.title_key,
|
||||
provider,
|
||||
now(),
|
||||
&episodes,
|
||||
enrichment,
|
||||
);
|
||||
if let Err(error) = stored {
|
||||
log::warn!("store episodes for {title_key}: {error}");
|
||||
log::warn!("store enrichment for {}: {error}", outcome.title_key);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2340,14 +2301,6 @@ struct CachedLibrary {
|
||||
titles: Vec<TitleRecord>,
|
||||
}
|
||||
|
||||
/// One title's share of an enrichment pass.
|
||||
enum Work {
|
||||
/// Search for it, and take its episodes if it matches.
|
||||
Match(Title),
|
||||
/// It is already matched; only the episode list is missing.
|
||||
Episodes(Title, Box<TitleMetadata>),
|
||||
}
|
||||
|
||||
/// How many provider lookups may be in flight at once.
|
||||
///
|
||||
/// Deliberately small. Both providers rate-limit by client, and a phone
|
||||
|
||||
@@ -1903,44 +1903,18 @@ impl Engine {
|
||||
};
|
||||
let provider = service.provider();
|
||||
|
||||
let (stored, listed) = {
|
||||
let (stored, enriched) = {
|
||||
let catalog = self.catalog.lock();
|
||||
match catalog
|
||||
.all_metadata()
|
||||
.and_then(|stored| Ok((stored, catalog.episode_metadata_ages()?)))
|
||||
.and_then(|stored| Ok((stored, catalog.enrichment_ages()?)))
|
||||
{
|
||||
Ok(pair) => pair,
|
||||
Err(error) => return self.fail("read stored metadata", error),
|
||||
}
|
||||
};
|
||||
|
||||
// Two kinds of work, and the difference is a request saved: a title
|
||||
// that already has a good match but no episode list needs the episode
|
||||
// request only — searching for it again would ask the provider
|
||||
// something it has already answered.
|
||||
let mut pending: Vec<Work> = Vec::new();
|
||||
for title in titles {
|
||||
let record = stored.get(&title.key);
|
||||
// A hand-picked entry is never searched for again, not even by
|
||||
// "match again" — that button means "the automatic answers are
|
||||
// wrong", and re-deciding the one title the viewer already fixed by
|
||||
// hand is the opposite of what they asked for. Its episode list is
|
||||
// still fetched below if it is missing.
|
||||
let pinned = record.is_some_and(|record| record.manual && record.provider == provider);
|
||||
if !pinned && (force || !pstr_meta::service::is_usable(record, provider)) {
|
||||
pending.push(Work::Match(title));
|
||||
continue;
|
||||
}
|
||||
let has_episodes = listed
|
||||
.get(&title.key)
|
||||
.is_some_and(|(asked, _)| *asked == provider);
|
||||
if let Some(found) = record.and_then(|record| record.metadata.clone())
|
||||
&& !has_episodes
|
||||
{
|
||||
pending.push(Work::Episodes(title, Box::new(found)));
|
||||
}
|
||||
}
|
||||
|
||||
let pending = pstr_meta::service::plan(titles, &stored, &enriched, provider, force);
|
||||
if pending.is_empty() {
|
||||
return self.emit(Event::Status("everything is already matched".into()));
|
||||
}
|
||||
@@ -1959,25 +1933,7 @@ impl Engine {
|
||||
let Ok(_permit) = permits.acquire().await else {
|
||||
return None;
|
||||
};
|
||||
Some(match work {
|
||||
Work::Match(title) => match service.record(title).await {
|
||||
Ok(record) => {
|
||||
// Only for a title that matched: there is no id to
|
||||
// ask about otherwise.
|
||||
let episodes = match &record.metadata {
|
||||
Some(found) => service.title_episodes(title, found).await,
|
||||
None => Vec::new(),
|
||||
};
|
||||
Ok((record.title_key.clone(), Some(record), episodes))
|
||||
}
|
||||
Err(error) => Err(error),
|
||||
},
|
||||
Work::Episodes(title, found) => Ok((
|
||||
title.key.clone(),
|
||||
None,
|
||||
service.title_episodes(title, found).await,
|
||||
)),
|
||||
})
|
||||
Some(service.work(work).await)
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1985,8 +1941,8 @@ impl Engine {
|
||||
let mut episodes_found = 0usize;
|
||||
while let Some(result) = lookups.next().await {
|
||||
match result {
|
||||
Some(Ok((title_key, record, episodes))) => {
|
||||
if let Some(record) = &record {
|
||||
Some(Ok(outcome)) => {
|
||||
if let Some(record) = &outcome.record {
|
||||
if record.metadata.is_some() {
|
||||
matched += 1;
|
||||
} else {
|
||||
@@ -1998,16 +1954,16 @@ impl Engine {
|
||||
tracing::warn!("store metadata for {}: {error}", record.title_key);
|
||||
}
|
||||
}
|
||||
if !episodes.is_empty() {
|
||||
episodes_found += episodes.len();
|
||||
let stored = self.catalog.lock().set_episode_metadata(
|
||||
&title_key,
|
||||
if let Some(enrichment) = &outcome.enrichment {
|
||||
episodes_found += enrichment.episodes.len();
|
||||
let stored = self.catalog.lock().set_enrichment(
|
||||
&outcome.title_key,
|
||||
provider,
|
||||
now(),
|
||||
&episodes,
|
||||
enrichment,
|
||||
);
|
||||
if let Err(error) = stored {
|
||||
tracing::warn!("store episodes for {title_key}: {error}");
|
||||
tracing::warn!("store enrichment for {}: {error}", outcome.title_key);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2078,18 +2034,22 @@ impl Engine {
|
||||
return engine.fail("store the match", error);
|
||||
}
|
||||
|
||||
let episodes = service.title_episodes(&title, &found).await;
|
||||
if !episodes.is_empty() {
|
||||
let stored = engine.catalog.lock().set_episode_metadata(
|
||||
// A failure leaves the title un-enriched, and the next match run
|
||||
// asks again.
|
||||
match service.enrich(&title, &found).await {
|
||||
Ok(enrichment) => {
|
||||
let stored = engine.catalog.lock().set_enrichment(
|
||||
&title.key,
|
||||
service.provider(),
|
||||
now(),
|
||||
&episodes,
|
||||
&enrichment,
|
||||
);
|
||||
if let Err(error) = stored {
|
||||
tracing::warn!("store episodes for {}: {error}", title.key);
|
||||
tracing::warn!("store enrichment for {}: {error}", title.key);
|
||||
}
|
||||
}
|
||||
Err(error) => tracing::warn!("enrich {}: {error}", title.key),
|
||||
}
|
||||
|
||||
engine.emit(Event::Status(format!("{} is now {name}", title.name)));
|
||||
engine.load_metadata();
|
||||
@@ -2202,15 +2162,6 @@ fn digest(url: &str) -> String {
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The cache and texture key for a file's poster.
|
||||
/// One title's worth of work in a match run.
|
||||
enum Work {
|
||||
/// Search for it, and take its episodes if it matches.
|
||||
Match(Title),
|
||||
/// It is already matched; only the episode list is missing.
|
||||
Episodes(Title, Box<pstr_core::metadata::TitleMetadata>),
|
||||
}
|
||||
|
||||
/// Unix seconds, for the "when was this asked" columns.
|
||||
fn now() -> i64 {
|
||||
std::time::SystemTime::now()
|
||||
|
||||
@@ -369,7 +369,9 @@ fn metadata_form(
|
||||
"A share filled by the Proton Drive desktop client carries no thumbnails, so \
|
||||
without this every tile is a pair of initials. Turning it on sends the titles \
|
||||
in your library — not your files, and nothing about what you have watched — to \
|
||||
the provider you choose, over HTTPS, each time a new one appears.",
|
||||
the provider you choose, over HTTPS, each time a new one appears. With AniList, \
|
||||
the ids it matched — not the titles — then go to ani.zip for episode names and \
|
||||
wide artwork.",
|
||||
));
|
||||
ui.add_space(theme::space::L);
|
||||
|
||||
|
||||
+62
-22
@@ -483,18 +483,18 @@ async fn metadata(dirs: &AppDirs, command: MetadataCommand) -> Result<()> {
|
||||
)
|
||||
.context("build the metadata provider")?;
|
||||
|
||||
let catalog = Catalog::open(&dirs.catalog_db()).context("open catalog")?;
|
||||
let mut catalog = Catalog::open(&dirs.catalog_db()).context("open catalog")?;
|
||||
let library = Library::build(catalog.all_files()?, &catalog.all_watch_states()?);
|
||||
let stored = catalog.all_metadata()?;
|
||||
let enriched = catalog.enrichment_ages()?;
|
||||
|
||||
let mut pending: Vec<_> = library
|
||||
.titles
|
||||
.iter()
|
||||
.filter(|title| {
|
||||
force
|
||||
|| !pstr_meta::service::is_usable(stored.get(&title.key), settings.provider)
|
||||
})
|
||||
.collect();
|
||||
let mut pending = pstr_meta::service::plan(
|
||||
library.titles,
|
||||
&stored,
|
||||
&enriched,
|
||||
settings.provider,
|
||||
force,
|
||||
);
|
||||
if let Some(limit) = limit {
|
||||
pending.truncate(limit);
|
||||
}
|
||||
@@ -504,7 +504,7 @@ async fn metadata(dirs: &AppDirs, command: MetadataCommand) -> Result<()> {
|
||||
return Ok(());
|
||||
}
|
||||
println!(
|
||||
"matching {} titles against {}…",
|
||||
"looking up {} titles against {}…",
|
||||
pending.len(),
|
||||
settings.provider.label()
|
||||
);
|
||||
@@ -513,15 +513,25 @@ async fn metadata(dirs: &AppDirs, command: MetadataCommand) -> Result<()> {
|
||||
// and the GUI already has the parallel path. What this command is
|
||||
// for is reading the output.
|
||||
let (mut matched, mut failed) = (0usize, 0usize);
|
||||
for title in pending {
|
||||
match service.record(title).await {
|
||||
Ok(record) => {
|
||||
for work in pending {
|
||||
let name = &work.title().name;
|
||||
let outcome = match service.work(&work).await {
|
||||
Ok(outcome) => outcome,
|
||||
Err(error) => {
|
||||
failed += 1;
|
||||
// Not stored: a failure to ask is not an answer, and the
|
||||
// title has to stay askable. See `pstr_meta::service`.
|
||||
println!(" {name:<40} ! {error}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
if let Some(record) = &outcome.record {
|
||||
match &record.metadata {
|
||||
Some(found) => {
|
||||
matched += 1;
|
||||
println!(
|
||||
" {:<40} → {} ({})",
|
||||
title.name,
|
||||
" {name:<40} → {} ({})",
|
||||
found.name,
|
||||
found
|
||||
.year
|
||||
@@ -529,16 +539,39 @@ async fn metadata(dirs: &AppDirs, command: MetadataCommand) -> Result<()> {
|
||||
.unwrap_or_else(|| "-".into())
|
||||
);
|
||||
}
|
||||
None => println!(" {:<40} → no match", title.name),
|
||||
None => println!(" {name:<40} → no match"),
|
||||
}
|
||||
catalog.set_metadata(&record)?;
|
||||
catalog.set_metadata(record)?;
|
||||
}
|
||||
Err(error) => {
|
||||
failed += 1;
|
||||
// Not stored: a failure to ask is not an answer, and the
|
||||
// title has to stay askable. See `pstr_meta::service`.
|
||||
println!(" {:<40} ! {error}", title.name);
|
||||
match &outcome.enrichment {
|
||||
Some(enrichment) => {
|
||||
println!(
|
||||
" {:<40} {} episodes{}",
|
||||
"",
|
||||
enrichment.episodes.len(),
|
||||
if enrichment.backdrop_url.is_some() {
|
||||
", fanart"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
);
|
||||
catalog.set_enrichment(
|
||||
&outcome.title_key,
|
||||
settings.provider,
|
||||
unix_now(),
|
||||
enrichment,
|
||||
)?;
|
||||
}
|
||||
// Matched, but the episodes could not be asked for; the
|
||||
// next run asks again.
|
||||
None if outcome
|
||||
.record
|
||||
.as_ref()
|
||||
.is_some_and(|r| r.metadata.is_some()) =>
|
||||
{
|
||||
println!(" {:<40} ! episodes not fetched", "");
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
}
|
||||
println!("{matched} matched, {failed} could not be looked up");
|
||||
@@ -926,3 +959,10 @@ async fn play(
|
||||
drop(opened.library);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Unix seconds, for the "when was this asked" columns.
|
||||
fn unix_now() -> i64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map_or(0, |since| since.as_secs() as i64)
|
||||
}
|
||||
+379
-41
@@ -17,11 +17,13 @@ use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::library::TitleKind;
|
||||
use crate::metadata::{EpisodeGuide, EpisodeMetadata, MetadataRecord, ProviderId, TitleMetadata};
|
||||
use crate::metadata::{
|
||||
Enrichment, EpisodeGuide, EpisodeMetadata, MetadataRecord, ProviderId, TitleMetadata,
|
||||
};
|
||||
use crate::naming::{self, ParsedName};
|
||||
|
||||
/// Bump when adding a migration.
|
||||
pub const SCHEMA_VERSION: i64 = 8;
|
||||
pub const SCHEMA_VERSION: i64 = 9;
|
||||
|
||||
const MIGRATION_V1: &str = r#"
|
||||
CREATE TABLE nodes (
|
||||
@@ -153,6 +155,30 @@ ALTER TABLE title_track_prefs ADD COLUMN audio_title TEXT;
|
||||
ALTER TABLE title_track_prefs ADD COLUMN subtitle_title TEXT;
|
||||
"#;
|
||||
|
||||
/// Episodes numbered three ways, and a record of which titles were enriched.
|
||||
///
|
||||
/// A release numbers its files by TVDB season, by absolute number inside a
|
||||
/// season folder, or by AniList entry; an episode row now carries all three so
|
||||
/// `EpisodeGuide::get` can meet whichever the filename used. `enriched_at` is
|
||||
/// set once a matched title's episodes and art were asked for — empty answers
|
||||
/// included, which is what stops a film being re-asked on every run.
|
||||
///
|
||||
/// The deletes are a repair, not a reset. Every AniList episode row came from
|
||||
/// `streamingEpisodes` — partial, numbered however a streaming site numbered
|
||||
/// it, and in places a later season captioned with season one's names — and
|
||||
/// every AniList backdrop was a 4.75:1 banner drawn into a 16:9 tile. Both are
|
||||
/// wrong rather than stale, so they go; the matches stay, hand-picked ones
|
||||
/// included, and the next match run enriches them again. See
|
||||
/// `docs/METADATA.md`.
|
||||
const MIGRATION_V9: &str = r#"
|
||||
ALTER TABLE episode_metadata ADD COLUMN absolute INTEGER;
|
||||
ALTER TABLE episode_metadata ADD COLUMN entry_season INTEGER;
|
||||
ALTER TABLE episode_metadata ADD COLUMN entry_number INTEGER;
|
||||
ALTER TABLE title_metadata ADD COLUMN enriched_at INTEGER;
|
||||
DELETE FROM episode_metadata WHERE provider = 'anilist';
|
||||
UPDATE title_metadata SET backdrop_url = NULL WHERE provider = 'anilist';
|
||||
"#;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct OfflineFile {
|
||||
pub revision_id: String,
|
||||
@@ -332,6 +358,12 @@ impl Catalog {
|
||||
tx.pragma_update(None, "user_version", 8)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
if version < 9 {
|
||||
let tx = self.conn.transaction()?;
|
||||
tx.execute_batch(MIGRATION_V9)?;
|
||||
tx.pragma_update(None, "user_version", 9)?;
|
||||
tx.commit()?;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -699,8 +731,25 @@ impl Catalog {
|
||||
///
|
||||
/// A miss is stored too — see [`MetadataRecord::metadata`]. Storing only
|
||||
/// matches would leave every unmatched title asking again on every render.
|
||||
///
|
||||
/// Clears the title's enrichment record, so the answer is enriched again;
|
||||
/// see [`Catalog::set_enrichment`]. And when the answer is a different
|
||||
/// entry from the one stored — or none — the old entry's episode rows go
|
||||
/// at once: they are not about this one, and a front end that pins a match
|
||||
/// without enriching it must not show them in the meantime.
|
||||
pub fn set_metadata(&self, record: &MetadataRecord) -> Result<()> {
|
||||
let data = record.metadata.as_ref();
|
||||
self.conn.execute(
|
||||
"DELETE FROM episode_metadata WHERE title_key = ?1 AND NOT EXISTS (
|
||||
SELECT 1 FROM title_metadata
|
||||
WHERE title_key = ?1 AND provider = ?2 AND remote_id IS ?3
|
||||
)",
|
||||
params![
|
||||
record.title_key,
|
||||
record.provider.as_str(),
|
||||
data.map(|data| data.remote_id.as_str()),
|
||||
],
|
||||
)?;
|
||||
self.conn.execute(
|
||||
"INSERT INTO title_metadata (
|
||||
title_key, provider, matched, fetched_at, remote_id, name, original_name,
|
||||
@@ -723,7 +772,8 @@ impl Catalog {
|
||||
genres = excluded.genres,
|
||||
episodes = excluded.episodes,
|
||||
url = excluded.url,
|
||||
manual = excluded.manual",
|
||||
manual = excluded.manual,
|
||||
enriched_at = NULL",
|
||||
params![
|
||||
record.title_key,
|
||||
record.provider.as_str(),
|
||||
@@ -776,12 +826,7 @@ impl Catalog {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Replace everything stored about one title's episodes.
|
||||
///
|
||||
/// A replace rather than an upsert per row: a provider that has *dropped*
|
||||
/// an episode — a special that was recounted, a season renumbered — should
|
||||
/// leave nothing behind, and an episode list is small enough that the
|
||||
/// difference is a few hundred microseconds.
|
||||
/// Replace everything stored about one title's episodes, and nothing else.
|
||||
pub fn set_episode_metadata(
|
||||
&mut self,
|
||||
title_key: &str,
|
||||
@@ -790,31 +835,39 @@ impl Catalog {
|
||||
episodes: &[EpisodeMetadata],
|
||||
) -> Result<()> {
|
||||
let tx = self.conn.transaction()?;
|
||||
replace_episodes(&tx, title_key, provider, fetched_at, episodes)?;
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Store what enriching a matched title found, and that it was asked.
|
||||
///
|
||||
/// The episode list is replaced even when the new one is empty — a title
|
||||
/// re-pinned from a series to its film must not keep the series' episode
|
||||
/// names. The backdrop only changes when one was found. And the title is
|
||||
/// marked enriched either way, so a film with no episodes is not asked
|
||||
/// about again on every run; [`Catalog::set_metadata`] clears the mark when
|
||||
/// the match itself changes.
|
||||
pub fn set_enrichment(
|
||||
&mut self,
|
||||
title_key: &str,
|
||||
provider: ProviderId,
|
||||
fetched_at: i64,
|
||||
enrichment: &Enrichment,
|
||||
) -> Result<()> {
|
||||
let tx = self.conn.transaction()?;
|
||||
replace_episodes(&tx, title_key, provider, fetched_at, &enrichment.episodes)?;
|
||||
tx.execute(
|
||||
"DELETE FROM episode_metadata WHERE title_key = ?1",
|
||||
params![title_key],
|
||||
)?;
|
||||
{
|
||||
let mut insert = tx.prepare(
|
||||
"INSERT INTO episode_metadata (
|
||||
title_key, season, number, provider, fetched_at,
|
||||
name, overview, still_url, air_date
|
||||
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
|
||||
)?;
|
||||
for episode in episodes {
|
||||
insert.execute(params![
|
||||
"UPDATE title_metadata
|
||||
SET enriched_at = ?3, backdrop_url = COALESCE(?4, backdrop_url)
|
||||
WHERE title_key = ?1 AND provider = ?2",
|
||||
params![
|
||||
title_key,
|
||||
episode.season.map_or(ABSOLUTE_SEASON, i64::from),
|
||||
episode.number,
|
||||
provider.as_str(),
|
||||
fetched_at,
|
||||
episode.name.as_deref(),
|
||||
episode.overview.as_deref(),
|
||||
episode.still_url.as_deref(),
|
||||
episode.air_date.as_deref(),
|
||||
])?;
|
||||
}
|
||||
}
|
||||
enrichment.backdrop_url.as_deref(),
|
||||
],
|
||||
)?;
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -825,7 +878,8 @@ impl Catalog {
|
||||
/// season drawn a row at a time must not be a SQLite round-trip per row.
|
||||
pub fn all_episode_metadata(&self) -> Result<HashMap<String, EpisodeGuide>> {
|
||||
let mut statement = self.conn.prepare(
|
||||
"SELECT title_key, season, number, name, overview, still_url, air_date
|
||||
"SELECT title_key, season, number, name, overview, still_url, air_date,
|
||||
absolute, entry_season, entry_number
|
||||
FROM episode_metadata",
|
||||
)?;
|
||||
let rows = statement.query_map([], |row| {
|
||||
@@ -836,6 +890,9 @@ impl Catalog {
|
||||
EpisodeMetadata {
|
||||
season: (season != ABSOLUTE_SEASON).then_some(season as u32),
|
||||
number: row.get::<_, i64>(2)? as u32,
|
||||
absolute: row.get(7)?,
|
||||
entry_season: row.get(8)?,
|
||||
entry_number: row.get(9)?,
|
||||
name: row.get(3)?,
|
||||
overview: row.get(4)?,
|
||||
still_url: row.get(5)?,
|
||||
@@ -855,13 +912,13 @@ impl Catalog {
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Which titles already have an episode list stored, and how fresh.
|
||||
///
|
||||
/// Enough to decide whether to ask again without reading every row: the
|
||||
/// answer is only ever compared against a TTL.
|
||||
pub fn episode_metadata_ages(&self) -> Result<HashMap<String, (ProviderId, i64)>> {
|
||||
/// When each matched title was last enriched, for the provider it is
|
||||
/// matched against. A title missing here has never been, or its match has
|
||||
/// changed since.
|
||||
pub fn enrichment_ages(&self) -> Result<HashMap<String, (ProviderId, i64)>> {
|
||||
let mut statement = self.conn.prepare(
|
||||
"SELECT title_key, provider, MIN(fetched_at) FROM episode_metadata GROUP BY title_key",
|
||||
"SELECT title_key, provider, enriched_at FROM title_metadata
|
||||
WHERE enriched_at IS NOT NULL",
|
||||
)?;
|
||||
let rows = statement.query_map([], |row| {
|
||||
Ok((
|
||||
@@ -873,15 +930,58 @@ impl Catalog {
|
||||
|
||||
let mut ages = HashMap::new();
|
||||
for row in rows {
|
||||
let (title_key, provider, fetched_at) = row?;
|
||||
let (title_key, provider, enriched_at) = row?;
|
||||
if let Some(provider) = ProviderId::parse(&provider) {
|
||||
ages.insert(title_key, (provider, fetched_at));
|
||||
ages.insert(title_key, (provider, enriched_at));
|
||||
}
|
||||
}
|
||||
Ok(ages)
|
||||
}
|
||||
}
|
||||
|
||||
/// Replace every episode row of one title with `episodes`, inside `tx`.
|
||||
///
|
||||
/// A replace rather than an upsert per row: a provider that has *dropped* an
|
||||
/// episode — a special that was recounted, a season renumbered — should leave
|
||||
/// nothing behind, and an episode list is small enough that the difference is a
|
||||
/// few hundred microseconds.
|
||||
fn replace_episodes(
|
||||
tx: &rusqlite::Transaction<'_>,
|
||||
title_key: &str,
|
||||
provider: ProviderId,
|
||||
fetched_at: i64,
|
||||
episodes: &[EpisodeMetadata],
|
||||
) -> Result<()> {
|
||||
tx.execute(
|
||||
"DELETE FROM episode_metadata WHERE title_key = ?1",
|
||||
params![title_key],
|
||||
)?;
|
||||
let mut insert = tx.prepare(
|
||||
"INSERT OR REPLACE INTO episode_metadata (
|
||||
title_key, season, number, provider, fetched_at,
|
||||
name, overview, still_url, air_date,
|
||||
absolute, entry_season, entry_number
|
||||
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12)",
|
||||
)?;
|
||||
for episode in episodes {
|
||||
insert.execute(params![
|
||||
title_key,
|
||||
episode.season.map_or(ABSOLUTE_SEASON, i64::from),
|
||||
episode.number,
|
||||
provider.as_str(),
|
||||
fetched_at,
|
||||
episode.name.as_deref(),
|
||||
episode.overview.as_deref(),
|
||||
episode.still_url.as_deref(),
|
||||
episode.air_date.as_deref(),
|
||||
episode.absolute,
|
||||
episode.entry_season,
|
||||
episode.entry_number,
|
||||
])?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `None` for a row whose provider column this build does not recognise.
|
||||
fn row_to_metadata(row: &rusqlite::Row<'_>) -> rusqlite::Result<Option<MetadataRecord>> {
|
||||
let title_key: String = row.get(0)?;
|
||||
@@ -1281,6 +1381,9 @@ mod tests {
|
||||
&[EpisodeMetadata {
|
||||
season: None,
|
||||
number: 1,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some("To You, in 2000 Years".into()),
|
||||
overview: None,
|
||||
still_url: None,
|
||||
@@ -1370,6 +1473,9 @@ mod tests {
|
||||
EpisodeMetadata {
|
||||
season: None,
|
||||
number: 57,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some("The Immortal Legion".into()),
|
||||
overview: Some("Ed and Al…".into()),
|
||||
still_url: None,
|
||||
@@ -1378,6 +1484,9 @@ mod tests {
|
||||
EpisodeMetadata {
|
||||
season: Some(1),
|
||||
number: 1,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some("Fullmetal Alchemist".into()),
|
||||
overview: None,
|
||||
still_url: Some("still.jpg".into()),
|
||||
@@ -1407,9 +1516,235 @@ mod tests {
|
||||
.expect("store again");
|
||||
let stored = catalog.all_episode_metadata().expect("read");
|
||||
assert_eq!(stored["fma"].len(), 1);
|
||||
}
|
||||
|
||||
let ages = catalog.episode_metadata_ages().expect("ages");
|
||||
assert_eq!(ages["fma"], (ProviderId::AniList, 200));
|
||||
/// The three numberings a release might use have to survive a restart, or
|
||||
/// an absolutely-numbered season folder stops finding its names.
|
||||
#[test]
|
||||
fn every_numbering_of_an_episode_round_trips() {
|
||||
let mut catalog = Catalog::in_memory().expect("open");
|
||||
let episode = EpisodeMetadata {
|
||||
season: Some(2),
|
||||
number: 11,
|
||||
absolute: Some(39),
|
||||
entry_season: Some(3),
|
||||
entry_number: Some(1),
|
||||
name: Some("Lightning's End".into()),
|
||||
overview: None,
|
||||
still_url: None,
|
||||
air_date: None,
|
||||
};
|
||||
catalog
|
||||
.set_episode_metadata(
|
||||
"mushishi",
|
||||
ProviderId::AniList,
|
||||
1,
|
||||
std::slice::from_ref(&episode),
|
||||
)
|
||||
.expect("store");
|
||||
let stored = catalog.all_episode_metadata().expect("read");
|
||||
assert_eq!(stored["mushishi"].get(Some(2), 11), Some(&episode));
|
||||
assert_eq!(stored["mushishi"].get(Some(3), 1), Some(&episode));
|
||||
}
|
||||
|
||||
/// **What left Jujutsu Kaisen 0 with the TV series' episode names.** The
|
||||
/// title was re-pinned to the film, the film had no episodes, and an empty
|
||||
/// list was never written — so the series' list stayed.
|
||||
#[test]
|
||||
fn an_empty_enrichment_clears_the_episodes_the_last_match_left() {
|
||||
let mut catalog = Catalog::in_memory().expect("open");
|
||||
let record = MetadataRecord {
|
||||
title_key: "jujutsu kaisen 0 movie".into(),
|
||||
provider: ProviderId::AniList,
|
||||
metadata: Some(found()),
|
||||
fetched_at: 1,
|
||||
manual: true,
|
||||
};
|
||||
catalog.set_metadata(&record).expect("store");
|
||||
catalog
|
||||
.set_episode_metadata(
|
||||
&record.title_key,
|
||||
ProviderId::AniList,
|
||||
1,
|
||||
&[EpisodeMetadata {
|
||||
season: Some(1),
|
||||
number: 1,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some("Ryomen Sukuna".into()),
|
||||
overview: None,
|
||||
still_url: None,
|
||||
air_date: None,
|
||||
}],
|
||||
)
|
||||
.expect("store episodes");
|
||||
|
||||
// Storing the same entry again keeps them; it is the same entry.
|
||||
catalog.set_metadata(&record).expect("store again");
|
||||
assert_eq!(catalog.all_episode_metadata().expect("read").len(), 1);
|
||||
|
||||
catalog
|
||||
.set_enrichment(
|
||||
&record.title_key,
|
||||
ProviderId::AniList,
|
||||
5,
|
||||
&Enrichment::default(),
|
||||
)
|
||||
.expect("enrich");
|
||||
assert!(catalog.all_episode_metadata().expect("read").is_empty());
|
||||
// Asked, and the answer was nothing: that is recorded, or a film is
|
||||
// asked again on every run.
|
||||
assert_eq!(
|
||||
catalog
|
||||
.enrichment_ages()
|
||||
.expect("ages")
|
||||
.get(&record.title_key),
|
||||
Some(&(ProviderId::AniList, 5))
|
||||
);
|
||||
}
|
||||
|
||||
/// Android pins a match without enriching it; until the next match run,
|
||||
/// the title must show no episode names rather than the old entry's.
|
||||
#[test]
|
||||
fn pinning_a_different_entry_drops_the_old_entrys_episodes_at_once() {
|
||||
let mut catalog = Catalog::in_memory().expect("open");
|
||||
let series = MetadataRecord {
|
||||
title_key: "jujutsu kaisen 0 movie".into(),
|
||||
provider: ProviderId::AniList,
|
||||
metadata: Some(found()),
|
||||
fetched_at: 1,
|
||||
manual: false,
|
||||
};
|
||||
catalog.set_metadata(&series).expect("store");
|
||||
catalog
|
||||
.set_episode_metadata(
|
||||
&series.title_key,
|
||||
ProviderId::AniList,
|
||||
1,
|
||||
&[EpisodeMetadata {
|
||||
season: Some(1),
|
||||
number: 1,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some("Ryomen Sukuna".into()),
|
||||
overview: None,
|
||||
still_url: None,
|
||||
air_date: None,
|
||||
}],
|
||||
)
|
||||
.expect("store episodes");
|
||||
|
||||
let film = MetadataRecord {
|
||||
metadata: Some(TitleMetadata {
|
||||
remote_id: "131573".into(),
|
||||
..found()
|
||||
}),
|
||||
manual: true,
|
||||
..series
|
||||
};
|
||||
catalog.set_metadata(&film).expect("pin");
|
||||
assert!(catalog.all_episode_metadata().expect("read").is_empty());
|
||||
}
|
||||
|
||||
/// Enrichment adds a backdrop when it found one and otherwise leaves the
|
||||
/// match's own alone; a new match makes the title un-enriched again.
|
||||
#[test]
|
||||
fn enrichment_keeps_the_backdrop_it_did_not_replace_and_a_new_match_resets_it() {
|
||||
let mut catalog = Catalog::in_memory().expect("open");
|
||||
let record = MetadataRecord {
|
||||
title_key: "attack on titan".into(),
|
||||
provider: ProviderId::AniList,
|
||||
metadata: Some(TitleMetadata {
|
||||
backdrop_url: Some("old.jpg".into()),
|
||||
..found()
|
||||
}),
|
||||
fetched_at: 1,
|
||||
manual: false,
|
||||
};
|
||||
catalog.set_metadata(&record).expect("store");
|
||||
let backdrop = |catalog: &Catalog| {
|
||||
catalog
|
||||
.metadata("attack on titan")
|
||||
.expect("read")
|
||||
.and_then(|record| record.metadata)
|
||||
.and_then(|found| found.backdrop_url)
|
||||
};
|
||||
|
||||
catalog
|
||||
.set_enrichment(
|
||||
"attack on titan",
|
||||
ProviderId::AniList,
|
||||
2,
|
||||
&Enrichment::default(),
|
||||
)
|
||||
.expect("enrich");
|
||||
assert_eq!(backdrop(&catalog).as_deref(), Some("old.jpg"));
|
||||
|
||||
let enrichment = Enrichment {
|
||||
backdrop_url: Some("fanart.jpg".into()),
|
||||
..Enrichment::default()
|
||||
};
|
||||
catalog
|
||||
.set_enrichment("attack on titan", ProviderId::AniList, 3, &enrichment)
|
||||
.expect("enrich again");
|
||||
assert_eq!(backdrop(&catalog).as_deref(), Some("fanart.jpg"));
|
||||
|
||||
catalog.set_metadata(&record).expect("match again");
|
||||
assert!(catalog.enrichment_ages().expect("ages").is_empty());
|
||||
}
|
||||
|
||||
/// Schema v9 repairs what AniList's first episode source wrote: its
|
||||
/// episode rows and its banners go, and the matches — hand-picked ones
|
||||
/// included — stay.
|
||||
#[test]
|
||||
fn upgrading_drops_anilists_episode_rows_and_banners_but_keeps_its_matches() {
|
||||
let conn = Connection::open_in_memory().expect("open");
|
||||
for migration in [
|
||||
MIGRATION_V1,
|
||||
MIGRATION_V2,
|
||||
MIGRATION_V3,
|
||||
MIGRATION_V4,
|
||||
MIGRATION_V5,
|
||||
MIGRATION_V6,
|
||||
MIGRATION_V7,
|
||||
MIGRATION_V8,
|
||||
] {
|
||||
conn.execute_batch(migration).expect("migrate");
|
||||
}
|
||||
conn.pragma_update(None, "user_version", 8)
|
||||
.expect("set version");
|
||||
conn.execute_batch(
|
||||
"INSERT INTO title_metadata (title_key, provider, matched, fetched_at, remote_id,
|
||||
name, kind, backdrop_url, manual)
|
||||
VALUES ('aot', 'anilist', 1, 1, '16498', 'Attack on Titan', 'series',
|
||||
'banner.jpg', 1),
|
||||
('got', 'tmdb', 1, 1, '1399', 'Game of Thrones', 'series',
|
||||
'backdrop.jpg', 0);
|
||||
INSERT INTO episode_metadata (title_key, season, number, provider, fetched_at, name)
|
||||
VALUES ('aot', 2, 1, 'anilist', 1, 'Season one''s first, wrongly'),
|
||||
('got', 1, 1, 'tmdb', 1, 'Winter Is Coming');",
|
||||
)
|
||||
.expect("seed");
|
||||
|
||||
let catalog = Catalog::from_connection(conn).expect("upgrade");
|
||||
let records = catalog.all_metadata().expect("read");
|
||||
let aot = records["aot"].metadata.as_ref().expect("still matched");
|
||||
assert!(records["aot"].manual);
|
||||
assert_eq!(aot.backdrop_url, None);
|
||||
assert_eq!(
|
||||
records["got"]
|
||||
.metadata
|
||||
.as_ref()
|
||||
.and_then(|got| got.backdrop_url.as_deref()),
|
||||
Some("backdrop.jpg")
|
||||
);
|
||||
|
||||
let episodes = catalog.all_episode_metadata().expect("read");
|
||||
assert!(!episodes.contains_key("aot"));
|
||||
assert_eq!(episodes["got"].len(), 1);
|
||||
assert!(catalog.enrichment_ages().expect("ages").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1423,6 +1758,9 @@ mod tests {
|
||||
&[EpisodeMetadata {
|
||||
season: None,
|
||||
number: 1,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some("Fullmetal Alchemist".into()),
|
||||
overview: None,
|
||||
still_url: None,
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
//! thumbnails at upload time and that client attaches none. See
|
||||
//! `docs/DEVELOPMENT.md`.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
@@ -62,7 +62,10 @@ impl ProviderId {
|
||||
/// One line on what it covers and what it costs to use.
|
||||
pub fn description(self) -> &'static str {
|
||||
match self {
|
||||
Self::AniList => "Anime, with no API key and no account.",
|
||||
Self::AniList => {
|
||||
"Anime, with no API key and no account. Episode names and wide art come from \
|
||||
ani.zip, which is sent the matched ids."
|
||||
}
|
||||
Self::Tmdb => "Film and television. Needs a free API key from themoviedb.org.",
|
||||
}
|
||||
}
|
||||
@@ -139,6 +142,23 @@ pub struct EpisodeMetadata {
|
||||
/// anime releases name their files.
|
||||
pub season: Option<u32>,
|
||||
pub number: u32,
|
||||
/// Its place counting straight through every season, when the provider
|
||||
/// knows it. What an absolutely-numbered file inside a season folder —
|
||||
/// `Attack on Titan/Season 2/… - 26.mkv` — is matched on.
|
||||
#[serde(default)]
|
||||
pub absolute: Option<u32>,
|
||||
/// Its season and number counting each provider *entry* as a season: the
|
||||
/// position of the entry it belongs to among the franchise's series
|
||||
/// entries, and its number within that entry.
|
||||
///
|
||||
/// Not TVDB's season. A release that follows AniList rather than TVDB
|
||||
/// names Mushishi's *Zoku Shou 2* `S03`, where TVDB files it as the second
|
||||
/// half of season two; this is what that file is answered from. See
|
||||
/// [`EpisodeGuide::get`] for when it is allowed to be.
|
||||
#[serde(default)]
|
||||
pub entry_season: Option<u32>,
|
||||
#[serde(default)]
|
||||
pub entry_number: Option<u32>,
|
||||
pub name: Option<String>,
|
||||
pub overview: Option<String>,
|
||||
/// A frame from the episode, around 16:9.
|
||||
@@ -149,6 +169,20 @@ pub struct EpisodeMetadata {
|
||||
pub air_date: Option<String>,
|
||||
}
|
||||
|
||||
/// What enriching a matched title found: its episodes, and art the search
|
||||
/// answer did not carry.
|
||||
///
|
||||
/// Stored as one unit by `Catalog::set_enrichment`, and recorded as asked even
|
||||
/// when both are empty — a film has no episode list, and without the record it
|
||||
/// would be asked again on every match run.
|
||||
#[derive(Debug, Clone, Default, PartialEq)]
|
||||
pub struct Enrichment {
|
||||
pub episodes: Vec<EpisodeMetadata>,
|
||||
/// A 16:9 picture of the whole title, when one was found. `None` leaves
|
||||
/// whatever backdrop the match already had.
|
||||
pub backdrop_url: Option<String>,
|
||||
}
|
||||
|
||||
/// Every episode a provider listed for one title, ready to look up.
|
||||
///
|
||||
/// The lookup is the whole reason this is a type rather than a `Vec`: a file
|
||||
@@ -160,16 +194,33 @@ pub struct EpisodeGuide {
|
||||
/// Keyed `(season, number)`, where `None` is the provider's own absolute
|
||||
/// numbering.
|
||||
entries: HashMap<(Option<u32>, u32), EpisodeMetadata>,
|
||||
/// [`EpisodeMetadata::absolute`] to the key of the entry holding it.
|
||||
absolute: HashMap<u32, (Option<u32>, u32)>,
|
||||
/// [`EpisodeMetadata::entry_season`] and `entry_number` to the key of the
|
||||
/// entry holding them.
|
||||
by_entry: HashMap<(u32, u32), (Option<u32>, u32)>,
|
||||
/// Every season some entry is filed under, for deciding when the entry
|
||||
/// numbering may answer.
|
||||
stated: HashSet<u32>,
|
||||
}
|
||||
|
||||
impl EpisodeGuide {
|
||||
pub fn new(episodes: Vec<EpisodeMetadata>) -> Self {
|
||||
Self {
|
||||
entries: episodes
|
||||
.into_iter()
|
||||
.map(|episode| ((episode.season, episode.number), episode))
|
||||
.collect(),
|
||||
let mut guide = Self::default();
|
||||
for episode in episodes {
|
||||
let key = (episode.season, episode.number);
|
||||
if let Some(absolute) = episode.absolute {
|
||||
guide.absolute.insert(absolute, key);
|
||||
}
|
||||
if let (Some(season), Some(number)) = (episode.entry_season, episode.entry_number) {
|
||||
guide.by_entry.insert((season, number), key);
|
||||
}
|
||||
if let Some(season) = episode.season {
|
||||
guide.stated.insert(season);
|
||||
}
|
||||
guide.entries.insert(key, episode);
|
||||
}
|
||||
guide
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
@@ -182,20 +233,25 @@ impl EpisodeGuide {
|
||||
|
||||
/// What the provider says about the file numbered like this.
|
||||
///
|
||||
/// Tried in order: the season and number the filename states; the
|
||||
/// provider's absolute numbering; and, for a file with no season at all,
|
||||
/// season one. That second fallback is what makes an absolutely-numbered
|
||||
/// release line up with a provider that files everything under a single
|
||||
/// season — the common case for anime on TMDB.
|
||||
/// Tried in order:
|
||||
///
|
||||
/// **The absolute fallback stops at season one.** A provider that numbers
|
||||
/// straight through is one that split nothing, so its episode seven is the
|
||||
/// seventh of the show — which is season one's, not season two's. Answering
|
||||
/// `S02E07` with it captions every episode of every later season with the
|
||||
/// wrong name, and a wrong caption does not look like a bug: it looks like
|
||||
/// the library is wrong. Season two upwards is answered from an entry that
|
||||
/// states that season or not at all — see
|
||||
/// `Provider::seasons_are_separate_entries`.
|
||||
/// 1. the season and number the filename states;
|
||||
/// 2. for a file in no season or in season one, the provider's own absolute
|
||||
/// list, and for a file in no season, season one — what makes `#057`
|
||||
/// line up with TMDB filing a 64-episode anime under one season;
|
||||
/// 3. the absolute number, into the season the file states — what makes
|
||||
/// `Season 2/… - 26.mkv` the second season's first episode;
|
||||
/// 4. the entry numbering, for a season the provider files nothing under.
|
||||
///
|
||||
/// **No fallback crosses into another season.** Step 2 stops at season one
|
||||
/// and step 3 only answers inside the season the file named, because
|
||||
/// answering `S02E07` with season one's seventh episode captions every
|
||||
/// episode of every later season with the wrong name — and a wrong caption
|
||||
/// does not look like a bug, it looks like the library is wrong. Step 4
|
||||
/// only answers a season the provider has nothing filed under, so a release
|
||||
/// numbered the provider's way is never re-read the other way: Mushoku
|
||||
/// Tensei's second *entry* is the second half of season one, and its
|
||||
/// `S02E01` is season two's.
|
||||
pub fn get(&self, season: Option<u32>, number: u32) -> Option<&EpisodeMetadata> {
|
||||
if let Some(found) = self.entries.get(&(season, number)) {
|
||||
return Some(found);
|
||||
@@ -205,14 +261,31 @@ impl EpisodeGuide {
|
||||
{
|
||||
return Some(found);
|
||||
}
|
||||
season
|
||||
.is_none()
|
||||
.then(|| self.entries.get(&(Some(1), number)))?
|
||||
if season.is_none()
|
||||
&& let Some(found) = self.entries.get(&(Some(1), number))
|
||||
{
|
||||
return Some(found);
|
||||
}
|
||||
if let Some(found) = self
|
||||
.absolute
|
||||
.get(&number)
|
||||
.and_then(|key| self.entries.get(key))
|
||||
&& (season.is_none() || found.season == season)
|
||||
{
|
||||
return Some(found);
|
||||
}
|
||||
let season = season?;
|
||||
if self.stated.contains(&season) {
|
||||
return None;
|
||||
}
|
||||
self.by_entry
|
||||
.get(&(season, number))
|
||||
.and_then(|key| self.entries.get(key))
|
||||
}
|
||||
|
||||
/// Which seasons this guide actually has answers for. `None` is the
|
||||
/// provider's own absolute numbering.
|
||||
pub fn seasons(&self) -> std::collections::HashSet<Option<u32>> {
|
||||
pub fn seasons(&self) -> HashSet<Option<u32>> {
|
||||
self.entries
|
||||
.values()
|
||||
.map(|episode| episode.season)
|
||||
@@ -335,6 +408,9 @@ mod tests {
|
||||
EpisodeMetadata {
|
||||
season,
|
||||
number,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: Some(name.into()),
|
||||
overview: None,
|
||||
still_url: None,
|
||||
@@ -342,6 +418,89 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// An episode as ani.zip describes one: TVDB's season and number, TVDB's
|
||||
/// absolute number, and its place in the AniList entry it came from.
|
||||
fn numbered(
|
||||
(season, number): (u32, u32),
|
||||
absolute: u32,
|
||||
(entry_season, entry_number): (u32, u32),
|
||||
name: &str,
|
||||
) -> EpisodeMetadata {
|
||||
EpisodeMetadata {
|
||||
absolute: Some(absolute),
|
||||
entry_season: Some(entry_season),
|
||||
entry_number: Some(entry_number),
|
||||
..episode(Some(season), number, name)
|
||||
}
|
||||
}
|
||||
|
||||
fn name_of(guide: &EpisodeGuide, season: Option<u32>, number: u32) -> Option<String> {
|
||||
guide.get(season, number).and_then(|e| e.name.clone())
|
||||
}
|
||||
|
||||
/// `Attack on Titan/Season 2/[Anime Time] Attack On Titan - 26.mkv`: a
|
||||
/// season folder full of absolutely-numbered files.
|
||||
#[test]
|
||||
fn an_absolute_number_in_a_season_folder_finds_that_seasons_episode() {
|
||||
let guide = EpisodeGuide::new(vec![
|
||||
numbered((1, 1), 1, (1, 1), "To You, in 2000 Years"),
|
||||
numbered((2, 1), 26, (2, 1), "Beast Titan"),
|
||||
]);
|
||||
assert_eq!(name_of(&guide, Some(2), 26).as_deref(), Some("Beast Titan"));
|
||||
// And with no season at all, the absolute number is the whole answer.
|
||||
assert_eq!(name_of(&guide, None, 26).as_deref(), Some("Beast Titan"));
|
||||
}
|
||||
|
||||
/// The absolute fallback must not carry a file into a season it did not
|
||||
/// name: `S03E13` of a show whose 13th episode is in season one is not that
|
||||
/// episode.
|
||||
#[test]
|
||||
fn an_absolute_number_never_answers_for_another_season() {
|
||||
let guide = EpisodeGuide::new(vec![
|
||||
numbered((1, 13), 13, (1, 13), "Season one's thirteenth"),
|
||||
numbered((3, 1), 38, (3, 1), "Smoke Signal"),
|
||||
]);
|
||||
assert_eq!(guide.get(Some(3), 13), None);
|
||||
}
|
||||
|
||||
/// Mushishi: TVDB files both halves of *Zoku Shou* as season two, and the
|
||||
/// release calls the second half `S03`. Season three exists nowhere at the
|
||||
/// provider, so the entry numbering answers it.
|
||||
#[test]
|
||||
fn a_season_the_provider_does_not_have_is_answered_by_entry() {
|
||||
let guide = EpisodeGuide::new(vec![
|
||||
numbered((1, 1), 1, (1, 1), "The Green Seat"),
|
||||
numbered((2, 1), 28, (2, 1), "The Hand That Caresses the Night"),
|
||||
numbered((2, 11), 39, (3, 1), "Lightning's End"),
|
||||
]);
|
||||
assert_eq!(
|
||||
name_of(&guide, Some(3), 1).as_deref(),
|
||||
Some("Lightning's End")
|
||||
);
|
||||
assert_eq!(
|
||||
name_of(&guide, Some(2), 1).as_deref(),
|
||||
Some("The Hand That Caresses the Night")
|
||||
);
|
||||
}
|
||||
|
||||
/// Mushoku Tensei: AniList's second entry is the second half of TVDB's
|
||||
/// season one. A release numbered TVDB's way names season two `S02`, and
|
||||
/// the entry numbering must not answer it with that second half.
|
||||
#[test]
|
||||
fn the_entry_numbering_never_overrides_a_season_the_provider_has() {
|
||||
let guide = EpisodeGuide::new(vec![
|
||||
numbered((1, 12), 12, (2, 1), "Cour two's first"),
|
||||
numbered((2, 1), 26, (3, 1), "Season two's first"),
|
||||
]);
|
||||
assert_eq!(
|
||||
name_of(&guide, Some(2), 1).as_deref(),
|
||||
Some("Season two's first")
|
||||
);
|
||||
// A second-season episode the provider does not list is unanswered,
|
||||
// not answered from the entry numbering.
|
||||
assert_eq!(guide.get(Some(2), 2), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_file_that_states_its_season_is_answered_from_that_season() {
|
||||
let guide = EpisodeGuide::new(vec![
|
||||
|
||||
+259
-150
@@ -5,14 +5,22 @@
|
||||
//! also cost them a signup. The trade is coverage — AniList is anime and nothing
|
||||
//! else, so a library of films wants [`crate::tmdb`].
|
||||
//!
|
||||
//! One query shape is used for everything: a `Page` of `media` matching the
|
||||
//! One query shape is used for every search: a `Page` of `media` matching the
|
||||
//! search string, with every name AniList knows the show by. The alias list is
|
||||
//! the whole reason this works — see [`crate::matching`].
|
||||
//!
|
||||
//! AniList identifies a title and keeps nothing worth having about its
|
||||
//! episodes, so enriching a match leans on [`crate::anizip`] for those, walking
|
||||
//! AniList's own `SEQUEL` relations to find every entry the library's seasons
|
||||
//! are spread across.
|
||||
|
||||
use pstr_core::library::TitleKind;
|
||||
use pstr_core::metadata::{EpisodeMetadata, ProviderId, TitleMetadata};
|
||||
use std::collections::HashSet;
|
||||
|
||||
use pstr_core::library::{self, TitleKind};
|
||||
use pstr_core::metadata::{Enrichment, EpisodeGuide, EpisodeMetadata, ProviderId, TitleMetadata};
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::anizip::AniZip;
|
||||
use crate::error::{Error, Result};
|
||||
use crate::limiter::RateLimiter;
|
||||
use crate::matching::{Candidate, Query};
|
||||
@@ -56,7 +64,6 @@ query ($search: String, $perPage: Int) {
|
||||
description(asHtml: false)
|
||||
startDate { year }
|
||||
coverImage { extraLarge large }
|
||||
bannerImage
|
||||
averageScore
|
||||
genres
|
||||
episodes
|
||||
@@ -67,31 +74,40 @@ query ($search: String, $perPage: Int) {
|
||||
}
|
||||
"#;
|
||||
|
||||
/// Episode titles, such as AniList has them.
|
||||
/// The entries an entry leads on to.
|
||||
///
|
||||
/// `streamingEpisodes` is the only place AniList keeps them — there is no
|
||||
/// per-episode overview in its API at all — and the entries are what the
|
||||
/// streaming sites published: `"Episode 57 - The Immortal Legion"`, in airing
|
||||
/// order, occasionally with a gap. So the number is read out of the title where
|
||||
/// it says one, and only falls back to the position in the list.
|
||||
const EPISODES: &str = r#"
|
||||
/// One hop at a time rather than a nested query several levels deep: most
|
||||
/// titles need no hop at all, and the ones that do stop as soon as the
|
||||
/// library's files are answered.
|
||||
const RELATIONS: &str = r#"
|
||||
query ($id: Int) {
|
||||
Media(id: $id, type: ANIME) {
|
||||
episodes
|
||||
streamingEpisodes { title thumbnail }
|
||||
relations { edges { relationType node { id type format } } }
|
||||
}
|
||||
}
|
||||
"#;
|
||||
|
||||
/// How many entries of one franchise to read before giving up on files that
|
||||
/// still have no answer.
|
||||
///
|
||||
/// Attack on Titan's four seasons are seven entries and Bleach's seventeen are
|
||||
/// five, so twelve covers every long run in a real library while bounding what
|
||||
/// a file no entry will ever answer — a special, a recap — can cost.
|
||||
const MAX_CHAIN_ENTRIES: usize = 12;
|
||||
|
||||
pub struct AniList {
|
||||
http: reqwest::Client,
|
||||
/// Shared by every lookup in flight — see [`crate::limiter`].
|
||||
limiter: RateLimiter,
|
||||
anizip: AniZip,
|
||||
}
|
||||
|
||||
impl AniList {
|
||||
pub fn new(http: reqwest::Client) -> Self {
|
||||
/// `language` is the one episode names are wanted in, where ani.zip has
|
||||
/// them in it.
|
||||
pub fn new(http: reqwest::Client, language: String) -> Self {
|
||||
Self {
|
||||
anizip: AniZip::new(http.clone(), language),
|
||||
http,
|
||||
limiter: RateLimiter::per_minute(REQUESTS_PER_MINUTE),
|
||||
}
|
||||
@@ -103,11 +119,6 @@ impl Provider for AniList {
|
||||
ProviderId::AniList
|
||||
}
|
||||
|
||||
/// A sequel is its own entry here, numbering its episodes from one.
|
||||
fn seasons_are_separate_entries(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
async fn search(&self, query: &Query) -> Result<Vec<Candidate>> {
|
||||
// Each term is asked only because the one before it came back with
|
||||
// nothing at all — see `search_terms`. A page of candidates, however
|
||||
@@ -133,29 +144,92 @@ impl Provider for AniList {
|
||||
Ok(Vec::new())
|
||||
}
|
||||
|
||||
async fn episodes(&self, title: &TitleMetadata) -> Result<Vec<EpisodeMetadata>> {
|
||||
/// The episodes of every entry the library's files are spread across, and
|
||||
/// the series' fanart.
|
||||
///
|
||||
/// AniList files a sequel as its own entry — Bleach's seventeenth season is
|
||||
/// four of them, Mushoku Tensei's first is two — so a title's match is
|
||||
/// only where its episodes *start*. From there the `SEQUEL` relations are
|
||||
/// followed one entry at a time, each entry's episodes taken from ani.zip,
|
||||
/// until every numbered file in `title` has an answer, the franchise ends,
|
||||
/// or [`MAX_CHAIN_ENTRIES`] is reached. A one-season show is one ani.zip
|
||||
/// request and no relations query at all.
|
||||
///
|
||||
/// Following relations rather than searching for `<title> 2nd Season` is
|
||||
/// the point: that search found the base entry again for Attack on Titan,
|
||||
/// whose seasons two to four were then captioned with season one's names.
|
||||
async fn enrich(&self, title: &library::Title, found: &TitleMetadata) -> Result<Enrichment> {
|
||||
// The id came from our own search response, so a non-numeric one is a
|
||||
// record written by some other version — not worth an error, and there
|
||||
// is nothing to ask about.
|
||||
let Ok(id) = title.remote_id.parse::<i64>() else {
|
||||
return Ok(Vec::new());
|
||||
let Ok(first) = found.remote_id.parse::<i64>() else {
|
||||
return Ok(Enrichment::default());
|
||||
};
|
||||
let wanted: Vec<(Option<u32>, u32)> = title
|
||||
.episodes()
|
||||
.filter_map(|episode| Some((episode.node.parsed.season, episode.node.parsed.episode?)))
|
||||
.collect();
|
||||
|
||||
let data: Option<MediaData> = self
|
||||
.query(serde_json::json!({
|
||||
"query": EPISODES,
|
||||
"variables": { "id": id },
|
||||
}))
|
||||
.await?;
|
||||
let mut enrichment = Enrichment::default();
|
||||
let mut taken = HashSet::new();
|
||||
let mut visited = HashSet::new();
|
||||
let mut series_entries = 0;
|
||||
let mut next = Some(Link {
|
||||
id: first,
|
||||
series: found.kind == TitleKind::Series,
|
||||
});
|
||||
while let Some(link) = next.take() {
|
||||
visited.insert(link.id);
|
||||
let entry_season = link.series.then(|| {
|
||||
series_entries += 1;
|
||||
series_entries
|
||||
});
|
||||
let mapping = self.anizip.mapping(link.id, entry_season).await?;
|
||||
if enrichment.backdrop_url.is_none() {
|
||||
enrichment.backdrop_url = mapping.fanart;
|
||||
}
|
||||
// Where two entries both list an episode, the earlier one — the
|
||||
// one closer to the match — keeps it.
|
||||
enrichment.episodes.extend(
|
||||
mapping
|
||||
.episodes
|
||||
.into_iter()
|
||||
.filter(|episode| taken.insert((episode.season, episode.number))),
|
||||
);
|
||||
|
||||
Ok(data
|
||||
.and_then(|data| data.media)
|
||||
.map(|media| media.into_episodes())
|
||||
.unwrap_or_default())
|
||||
if answers_all(&enrichment.episodes, &wanted) || visited.len() >= MAX_CHAIN_ENTRIES {
|
||||
break;
|
||||
}
|
||||
next = self.sequel(link.id, &visited).await?;
|
||||
}
|
||||
|
||||
tracing::debug!(
|
||||
"{}: {} episodes over {} entries",
|
||||
found.name,
|
||||
enrichment.episodes.len(),
|
||||
visited.len()
|
||||
);
|
||||
Ok(enrichment)
|
||||
}
|
||||
}
|
||||
|
||||
impl AniList {
|
||||
/// The entry `id` continues into, if any.
|
||||
async fn sequel(&self, id: i64, visited: &HashSet<i64>) -> Result<Option<Link>> {
|
||||
let data: Option<RelationsData> = self
|
||||
.query(serde_json::json!({
|
||||
"query": RELATIONS,
|
||||
"variables": { "id": id },
|
||||
}))
|
||||
.await?;
|
||||
let edges = data
|
||||
.and_then(|data| data.media)
|
||||
.map(|media| media.relations.edges)
|
||||
.unwrap_or_default();
|
||||
Ok(next_in_chain(edges, visited))
|
||||
}
|
||||
|
||||
/// One GraphQL requestimpl AniList {
|
||||
/// One GraphQL request, paced, with AniList's two ways of failing — an HTTP
|
||||
/// status and an `errors` array under a 200 — folded into one.
|
||||
///
|
||||
@@ -278,78 +352,78 @@ struct SearchData {
|
||||
page: Page,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct MediaData {
|
||||
#[serde(rename = "Media")]
|
||||
media: Option<EpisodeMedia>,
|
||||
/// One step along a franchise.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
struct Link {
|
||||
id: i64,
|
||||
/// Whether the entry is part of the series proper — TV, a short or a web
|
||||
/// series — rather than a film or a special, which no release counts as a
|
||||
/// season.
|
||||
series: bool,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct EpisodeMedia {
|
||||
#[serde(rename = "streamingEpisodes", default)]
|
||||
streaming_episodes: Vec<StreamingEpisode>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct StreamingEpisode {
|
||||
title: Option<String>,
|
||||
thumbnail: Option<String>,
|
||||
}
|
||||
|
||||
impl EpisodeMedia {
|
||||
fn into_episodes(self) -> Vec<EpisodeMetadata> {
|
||||
self.streaming_episodes
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
.map(|(index, episode)| {
|
||||
let raw = episode.title.unwrap_or_default();
|
||||
let (number, name) = split_numbering(&raw);
|
||||
EpisodeMetadata {
|
||||
// AniList counts straight through, with no seasons: a
|
||||
// sequel is a separate entry, not a second season.
|
||||
season: None,
|
||||
// The position is the fallback and not the answer: the
|
||||
// lists have gaps, and an off-by-one here would caption
|
||||
// every episode of a show with the next one's name.
|
||||
number: number.unwrap_or(index as u32 + 1),
|
||||
name: name.filter(|name| !name.is_empty()),
|
||||
// AniList has no per-episode synopsis to give.
|
||||
overview: None,
|
||||
still_url: episode.thumbnail,
|
||||
air_date: None,
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Split `"Episode 57 - The Immortal Legion"` into its number and its name.
|
||||
/// Which of an entry's relations the franchise continues through.
|
||||
///
|
||||
/// Both parts are optional: plenty of entries are just `"Episode 12"`, and a
|
||||
/// few are only a name.
|
||||
fn split_numbering(title: &str) -> (Option<u32>, Option<String>) {
|
||||
let trimmed = title.trim();
|
||||
let rest = trimmed
|
||||
.strip_prefix("Episode ")
|
||||
.or_else(|| trimmed.strip_prefix("episode "))
|
||||
.or_else(|| trimmed.strip_prefix("EP"))
|
||||
.or_else(|| trimmed.strip_prefix("E"));
|
||||
|
||||
let Some(rest) = rest else {
|
||||
return (None, Some(trimmed.to_string()).filter(|t| !t.is_empty()));
|
||||
};
|
||||
|
||||
let digits: String = rest.chars().take_while(|c| c.is_ascii_digit()).collect();
|
||||
if digits.is_empty() {
|
||||
return (None, Some(trimmed.to_string()));
|
||||
/// A sequel that is itself a series wins over one that is a film or a
|
||||
/// special, but a special is still followed when it is the only way on:
|
||||
/// Mushishi's first series leads to its second only through an OVA.
|
||||
fn next_in_chain(edges: Vec<RelationEdge>, visited: &HashSet<i64>) -> Option<Link> {
|
||||
edges
|
||||
.into_iter()
|
||||
.filter(|edge| edge.relation_type.as_deref() == Some("SEQUEL"))
|
||||
.filter_map(|edge| edge.node)
|
||||
// A sequel can be the manga it was adapted into; only anime has
|
||||
// episodes to number.
|
||||
.filter(|node| node.kind.as_deref() == Some("ANIME") && !visited.contains(&node.id))
|
||||
.map(|node| Link {
|
||||
id: node.id,
|
||||
series: is_series_format(node.format.as_deref()),
|
||||
})
|
||||
.min_by_key(|link| !link.series)
|
||||
}
|
||||
|
||||
let name = rest[digits.len()..]
|
||||
.trim_start()
|
||||
.trim_start_matches(['-', '–', ':'])
|
||||
.trim()
|
||||
.to_string();
|
||||
(digits.parse().ok(), Some(name).filter(|n| !n.is_empty()))
|
||||
fn is_series_format(format: Option<&str>) -> bool {
|
||||
matches!(format, Some("TV" | "TV_SHORT" | "ONA"))
|
||||
}
|
||||
|
||||
/// Whether every numbered file has an answer in `episodes`.
|
||||
fn answers_all(episodes: &[EpisodeMetadata], wanted: &[(Option<u32>, u32)]) -> bool {
|
||||
let guide = EpisodeGuide::new(episodes.to_vec());
|
||||
wanted
|
||||
.iter()
|
||||
.all(|&(season, number)| guide.get(season, number).is_some())
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RelationsData {
|
||||
#[serde(rename = "Media")]
|
||||
media: Option<RelationsMedia>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RelationsMedia {
|
||||
relations: Relations,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct Relations {
|
||||
#[serde(default)]
|
||||
edges: Vec<RelationEdge>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RelationEdge {
|
||||
#[serde(rename = "relationType")]
|
||||
relation_type: Option<String>,
|
||||
node: Option<RelationNode>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RelationNode {
|
||||
id: i64,
|
||||
#[serde(rename = "type")]
|
||||
kind: Option<String>,
|
||||
format: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
@@ -369,8 +443,6 @@ struct Media {
|
||||
start_date: Option<FuzzyDate>,
|
||||
#[serde(rename = "coverImage")]
|
||||
cover_image: Option<CoverImage>,
|
||||
#[serde(rename = "bannerImage")]
|
||||
banner_image: Option<String>,
|
||||
#[serde(rename = "averageScore")]
|
||||
average_score: Option<f32>,
|
||||
#[serde(default)]
|
||||
@@ -444,9 +516,11 @@ impl Media {
|
||||
year: self.start_date.and_then(|date| date.year),
|
||||
kind: kind_of(self.format.as_deref()),
|
||||
poster_url: cover,
|
||||
// Usually `null`. AniList only has banners for the better-known
|
||||
// shows, which is why a tile has to cope with a poster's shape.
|
||||
backdrop_url: self.banner_image,
|
||||
// Not `bannerImage`: that is a 4.75:1 strip, and drawn into a
|
||||
// 16:9 tile it is a blurred sliver of its own middle. The
|
||||
// backdrop is ani.zip's fanart, found when the match is
|
||||
// enriched.
|
||||
backdrop_url: None,
|
||||
rating: self.average_score.map(|score| score / SCORE_SCALE),
|
||||
genres: self.genres,
|
||||
episodes: self.episodes,
|
||||
@@ -594,60 +668,95 @@ mod tests {
|
||||
assert_eq!(candidate.metadata.poster_url.as_deref(), Some("big.jpg"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_episode_title_gives_up_its_number_and_its_name() {
|
||||
assert_eq!(
|
||||
split_numbering("Episode 57 - The Immortal Legion"),
|
||||
(Some(57), Some("The Immortal Legion".to_string()))
|
||||
);
|
||||
assert_eq!(split_numbering("Episode 12"), (Some(12), None));
|
||||
// Some entries are only a name, and some sites write it differently.
|
||||
assert_eq!(
|
||||
split_numbering("The Day of the Beginning"),
|
||||
(None, Some("The Day of the Beginning".to_string()))
|
||||
);
|
||||
assert_eq!(
|
||||
split_numbering("E3 – Whatever"),
|
||||
(Some(3), Some("Whatever".to_string()))
|
||||
);
|
||||
assert_eq!(split_numbering(""), (None, None));
|
||||
fn edge(relation: &str, id: i64, kind: &str, format: &str) -> RelationEdge {
|
||||
RelationEdge {
|
||||
relation_type: Some(relation.into()),
|
||||
node: Some(RelationNode {
|
||||
id,
|
||||
kind: Some(kind.into()),
|
||||
format: Some(format.into()),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/// Attack on Titan's relations: an adaptation, two compilation films, a
|
||||
/// prequel OVA and a spin-off around the one sequel that is the next season.
|
||||
#[test]
|
||||
fn episode_numbers_come_from_the_titles_and_not_from_the_order() {
|
||||
// AniList lists what streaming sites published, and those lists have
|
||||
// gaps. Numbering by position would caption episode 4 as episode 3.
|
||||
let media: EpisodeMedia = serde_json::from_str(
|
||||
r#"{"streamingEpisodes": [
|
||||
{"title": "Episode 1 - Fullmetal Alchemist", "thumbnail": "one.jpg"},
|
||||
{"title": "Episode 3 - City of Heresy", "thumbnail": null}
|
||||
]}"#,
|
||||
)
|
||||
.expect("parse");
|
||||
|
||||
let episodes = media.into_episodes();
|
||||
assert_eq!(episodes[0].number, 1);
|
||||
assert_eq!(episodes[0].name.as_deref(), Some("Fullmetal Alchemist"));
|
||||
assert_eq!(episodes[0].still_url.as_deref(), Some("one.jpg"));
|
||||
assert_eq!(episodes[1].number, 3);
|
||||
// AniList counts straight through; nothing here is a season.
|
||||
assert!(episodes.iter().all(|episode| episode.season.is_none()));
|
||||
fn the_chain_continues_through_the_sequel_and_nothing_else() {
|
||||
let edges = vec![
|
||||
edge("ADAPTATION", 53390, "MANGA", "MANGA"),
|
||||
edge("ALTERNATIVE", 20691, "ANIME", "MOVIE"),
|
||||
edge("PREQUEL", 20811, "ANIME", "OVA"),
|
||||
edge("SPIN_OFF", 21281, "ANIME", "TV"),
|
||||
edge("SEQUEL", 20958, "ANIME", "TV"),
|
||||
];
|
||||
assert_eq!(
|
||||
next_in_chain(edges, &HashSet::new()),
|
||||
Some(Link {
|
||||
id: 20958,
|
||||
series: true
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
/// Mushishi's first series leads on only through an OVA, which is followed
|
||||
/// — but not counted as a season of the series.
|
||||
#[test]
|
||||
fn an_untitled_list_still_numbers_its_episodes() {
|
||||
let media: EpisodeMedia = serde_json::from_str(
|
||||
r#"{"streamingEpisodes": [{"title": null, "thumbnail": null},
|
||||
{"title": "", "thumbnail": null}]}"#,
|
||||
)
|
||||
.expect("parse");
|
||||
let episodes = media.into_episodes();
|
||||
assert_eq!(episodes[0].number, 1);
|
||||
assert_eq!(episodes[1].number, 2);
|
||||
assert!(episodes.iter().all(|episode| episode.name.is_none()));
|
||||
fn a_special_is_followed_when_it_is_the_only_way_on() {
|
||||
let edges = vec![edge("SEQUEL", 20526, "ANIME", "SPECIAL")];
|
||||
assert_eq!(
|
||||
next_in_chain(edges, &HashSet::new()),
|
||||
Some(Link {
|
||||
id: 20526,
|
||||
series: false
|
||||
})
|
||||
);
|
||||
// With a series beside it, the series wins.
|
||||
let edges = vec![
|
||||
edge("SEQUEL", 20752, "ANIME", "SPECIAL"),
|
||||
edge("SEQUEL", 20751, "ANIME", "TV"),
|
||||
];
|
||||
assert_eq!(
|
||||
next_in_chain(edges, &HashSet::new()).map(|link| link.id),
|
||||
Some(20751)
|
||||
);
|
||||
}
|
||||
|
||||
/// Missing fields are the normal case for a recently added entry, and must
|
||||
/// A franchise whose relations loop must end rather than walk forever.
|
||||
#[test]
|
||||
fn an_entry_already_read_is_not_read_again() {
|
||||
let edges = vec![edge("SEQUEL", 1, "ANIME", "TV")];
|
||||
assert_eq!(next_in_chain(edges, &HashSet::from([1])), None);
|
||||
let manga = vec![edge("SEQUEL", 2, "MANGA", "MANGA")];
|
||||
assert_eq!(next_in_chain(manga, &HashSet::new()), None);
|
||||
}
|
||||
|
||||
/// The walk stops once every file is answered and not before: a season
|
||||
/// folder of absolute numbers counts as answered by its own season.
|
||||
#[test]
|
||||
fn the_walk_stops_when_every_numbered_file_has_an_answer() {
|
||||
let episode = |season, number, absolute| EpisodeMetadata {
|
||||
season: Some(season),
|
||||
number,
|
||||
absolute: Some(absolute),
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: None,
|
||||
overview: None,
|
||||
still_url: None,
|
||||
air_date: None,
|
||||
};
|
||||
let season_one = vec![episode(1, 1, 1), episode(1, 2, 2)];
|
||||
assert!(answers_all(&season_one, &[(Some(1), 1), (None, 2)]));
|
||||
assert!(!answers_all(&season_one, &[(Some(1), 1), (Some(2), 26)]));
|
||||
|
||||
let both = vec![episode(1, 1, 1), episode(2, 1, 26)];
|
||||
assert!(answers_all(&both, &[(Some(1), 1), (Some(2), 26)]));
|
||||
// A film: nothing numbered, nothing to walk for.
|
||||
assert!(answers_all(&[], &[]));
|
||||
}
|
||||
|
||||
/// Missing fields are the normal case /// Missing fields are the normal case for a recently added entry, and must
|
||||
/// not fail the whole page of results.
|
||||
#[test]
|
||||
fn an_entry_with_almost_nothing_in_it_still_parses() {
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
//! ani.zip: what AniList does not say about one of its own entries.
|
||||
//!
|
||||
//! AniList is where a title is *identified* — its search and its aliases are
|
||||
//! what make a romaji folder name match — but it keeps no episode list worth
|
||||
//! the name and no landscape art. ani.zip answers both, keyed by the AniList id
|
||||
//! the match already produced, so asking it sends nothing about the library the
|
||||
//! match did not already send. No key and no account, like AniList.
|
||||
//!
|
||||
//! The episode list is TVDB's, and that is the point of it: release groups
|
||||
//! number their files the way TVDB does — `Bleach - S17E01` is TVDB's season
|
||||
//! 17, not AniList's — and every episode here carries TVDB's season, number and
|
||||
//! absolute number together. See `docs/METADATA.md`.
|
||||
//!
|
||||
//! It is a community service rather than an institution, so everything it says
|
||||
//! is stored, and an entry it does not know (`404`) is an empty answer rather
|
||||
//! than a failure.
|
||||
|
||||
use pstr_core::metadata::EpisodeMetadata;
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::error::{Error, Result};
|
||||
use crate::limiter::RateLimiter;
|
||||
|
||||
const ENDPOINT: &str = "https://api.ani.zip/mappings";
|
||||
|
||||
/// ani.zip publishes no limit. One a second is far below anything a CDN-cached
|
||||
/// JSON endpoint notices, and a library's enrichment is a one-off burst.
|
||||
const REQUESTS_PER_MINUTE: u32 = 60;
|
||||
|
||||
/// TVDB's own placeholder for an episode it has not named yet.
|
||||
const UNNAMED: &str = "TBA";
|
||||
|
||||
/// What ani.zip knows about one AniList entry.
|
||||
#[derive(Debug, Default)]
|
||||
pub struct Mapping {
|
||||
pub episodes: Vec<EpisodeMetadata>,
|
||||
/// TVDB's fanart for the whole series: 16:9, usually 1920×1080.
|
||||
pub fanart: Option<String>,
|
||||
}
|
||||
|
||||
pub struct AniZip {
|
||||
http: reqwest::Client,
|
||||
/// The episode-title language tried first; English, then romaji, then
|
||||
/// Japanese after it.
|
||||
language: String,
|
||||
limiter: RateLimiter,
|
||||
}
|
||||
|
||||
impl AniZip {
|
||||
pub fn new(http: reqwest::Client, language: String) -> Self {
|
||||
Self {
|
||||
http,
|
||||
language,
|
||||
limiter: RateLimiter::per_minute(REQUESTS_PER_MINUTE),
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything ani.zip has for the AniList entry `anilist_id`.
|
||||
///
|
||||
/// `entry_season` is where this entry sits among the franchise's series
|
||||
/// entries — see [`EpisodeMetadata::entry_season`] — and `None` for a film
|
||||
/// or a special, whose episodes no release numbers that way.
|
||||
pub async fn mapping(&self, anilist_id: i64, entry_season: Option<u32>) -> Result<Mapping> {
|
||||
self.limiter.acquire().await;
|
||||
let response = self
|
||||
.http
|
||||
.get(ENDPOINT)
|
||||
.query(&[("anilist_id", anilist_id)])
|
||||
.send()
|
||||
.await?;
|
||||
let status = response.status();
|
||||
if status.as_u16() == 404 {
|
||||
return Ok(Mapping::default());
|
||||
}
|
||||
if status.as_u16() == 429 {
|
||||
self.limiter.throttled(response.headers()).await;
|
||||
return Err(Error::RateLimited);
|
||||
}
|
||||
if !status.is_success() {
|
||||
return Err(Error::Http(format!("ani.zip answered {status}")));
|
||||
}
|
||||
let payload: Payload = response.json().await?;
|
||||
Ok(payload.into_mapping(&self.language, entry_season))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Deserialize, Default)]
|
||||
struct Payload {
|
||||
#[serde(default)]
|
||||
episodes: std::collections::BTreeMap<String, Episode>,
|
||||
#[serde(default)]
|
||||
images: Vec<Image>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct Episode {
|
||||
#[serde(default, deserialize_with = "lenient_number")]
|
||||
season_number: Option<u32>,
|
||||
#[serde(default, deserialize_with = "lenient_number")]
|
||||
episode_number: Option<u32>,
|
||||
#[serde(default, deserialize_with = "lenient_number")]
|
||||
absolute_episode_number: Option<u32>,
|
||||
title: Option<std::collections::HashMap<String, Option<String>>>,
|
||||
/// TVDB's synopsis.
|
||||
overview: Option<String>,
|
||||
/// AniDB's, which is there when TVDB's is not.
|
||||
summary: Option<String>,
|
||||
image: Option<String>,
|
||||
air_date: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct Image {
|
||||
cover_type: Option<String>,
|
||||
url: Option<String>,
|
||||
}
|
||||
|
||||
impl Payload {
|
||||
fn into_mapping(self, language: &str, entry_season: Option<u32>) -> Mapping {
|
||||
let fanart = self
|
||||
.images
|
||||
.into_iter()
|
||||
.find(|image| image.cover_type.as_deref() == Some("Fanart"))
|
||||
.and_then(|image| image.url);
|
||||
|
||||
let episodes = self
|
||||
.episodes
|
||||
.into_iter()
|
||||
.filter_map(|(key, episode)| {
|
||||
// An episode TVDB has not placed has nothing a filename could
|
||||
// be matched against — a film's own entry is the usual case.
|
||||
let season = episode.season_number?;
|
||||
let number = episode.episode_number?;
|
||||
// Keys are the entry's own numbering: `"1"`, `"2"` for its
|
||||
// episodes and `"S1"` for its specials, which no release counts
|
||||
// as part of the season.
|
||||
let entry_number = entry_season.and(key.parse::<u32>().ok());
|
||||
Some(EpisodeMetadata {
|
||||
season: Some(season),
|
||||
number,
|
||||
absolute: episode.absolute_episode_number,
|
||||
entry_season: entry_number.and(entry_season),
|
||||
entry_number,
|
||||
name: episode
|
||||
.title
|
||||
.as_ref()
|
||||
.and_then(|titles| title_in(titles, language)),
|
||||
overview: text(episode.overview).or_else(|| text(episode.summary)),
|
||||
still_url: text(episode.image),
|
||||
air_date: text(episode.air_date),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
||||
Mapping { episodes, fanart }
|
||||
}
|
||||
}
|
||||
|
||||
/// The episode's name in `language`, else English, romaji or Japanese.
|
||||
fn title_in(
|
||||
titles: &std::collections::HashMap<String, Option<String>>,
|
||||
language: &str,
|
||||
) -> Option<String> {
|
||||
[language, "en", "x-jat", "ja"]
|
||||
.into_iter()
|
||||
.filter_map(|language| titles.get(language).cloned().flatten())
|
||||
.map(|title| title.trim().to_string())
|
||||
.find(|title| !title.is_empty() && title != UNNAMED)
|
||||
}
|
||||
|
||||
fn text(value: Option<String>) -> Option<String> {
|
||||
value
|
||||
.map(|value| value.trim().to_string())
|
||||
.filter(|value| !value.is_empty())
|
||||
}
|
||||
|
||||
/// A number, whether it arrives as one or as a string of digits.
|
||||
///
|
||||
/// ani.zip writes its regular episodes' numbers as JSON numbers and its
|
||||
/// specials' as strings; one strict field would fail the whole entry over the
|
||||
/// specials.
|
||||
fn lenient_number<'de, D>(deserializer: D) -> std::result::Result<Option<u32>, D::Error>
|
||||
where
|
||||
D: serde::Deserializer<'de>,
|
||||
{
|
||||
Ok(match serde_json::Value::deserialize(deserializer)? {
|
||||
serde_json::Value::Number(number) => number.as_u64().and_then(|n| u32::try_from(n).ok()),
|
||||
serde_json::Value::String(text) => text.trim().parse().ok(),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Attack on Titan Season 2 as ani.zip answers it, trimmed: a regular
|
||||
/// episode numbered in numbers, a special numbered in strings, a film-like
|
||||
/// entry TVDB never placed, and the series' art.
|
||||
const SEASON_TWO: &str = r#"{
|
||||
"episodes": {
|
||||
"1": {"seasonNumber": 2, "episodeNumber": 1, "absoluteEpisodeNumber": 26,
|
||||
"episode": "1",
|
||||
"title": {"en": "Beast Titan", "ja": "獣の巨人", "x-jat": "Kemono no Kyojin"},
|
||||
"overview": "Titans appear inside Wall Rose.",
|
||||
"image": "https://artworks.thetvdb.com/still.jpg",
|
||||
"airDate": "2017-04-01"},
|
||||
"S1": {"seasonNumber": "0", "episodeNumber": "7", "episode": "S1",
|
||||
"title": {"en": "TBA", "x-jat": "Ilse no Techou"},
|
||||
"summary": "An OVA."},
|
||||
"2": {"episode": "2", "title": {"en": "Unplaced"}}
|
||||
},
|
||||
"images": [
|
||||
{"coverType": "Banner", "url": "banner.jpg"},
|
||||
{"coverType": "Fanart", "url": "fanart.jpg"}
|
||||
]
|
||||
}"#;
|
||||
|
||||
fn mapping(entry_season: Option<u32>) -> Mapping {
|
||||
serde_json::from_str::<Payload>(SEASON_TWO)
|
||||
.expect("parse")
|
||||
.into_mapping("en", entry_season)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_episode_carries_tvdbs_numbering_and_its_entrys() {
|
||||
let mapping = mapping(Some(2));
|
||||
let first = mapping
|
||||
.episodes
|
||||
.iter()
|
||||
.find(|episode| episode.season == Some(2))
|
||||
.expect("the regular episode");
|
||||
assert_eq!(first.number, 1);
|
||||
assert_eq!(first.absolute, Some(26));
|
||||
assert_eq!((first.entry_season, first.entry_number), (Some(2), Some(1)));
|
||||
assert_eq!(first.name.as_deref(), Some("Beast Titan"));
|
||||
assert_eq!(
|
||||
first.overview.as_deref(),
|
||||
Some("Titans appear inside Wall Rose.")
|
||||
);
|
||||
assert_eq!(first.air_date.as_deref(), Some("2017-04-01"));
|
||||
}
|
||||
|
||||
/// A special is numbered in strings and filed under season zero, and no
|
||||
/// release counts it as an episode of the entry it came with.
|
||||
#[test]
|
||||
fn a_special_keeps_its_tvdb_place_and_takes_no_entry_number() {
|
||||
let mapping = mapping(Some(2));
|
||||
let special = mapping
|
||||
.episodes
|
||||
.iter()
|
||||
.find(|episode| episode.season == Some(0))
|
||||
.expect("the special");
|
||||
assert_eq!(special.number, 7);
|
||||
assert_eq!(special.entry_number, None);
|
||||
assert_eq!(special.entry_season, None);
|
||||
// `TBA` is TVDB saying it has no name yet, so the next language answers.
|
||||
assert_eq!(special.name.as_deref(), Some("Ilse no Techou"));
|
||||
// AniDB's summary stands in for TVDB's missing overview.
|
||||
assert_eq!(special.overview.as_deref(), Some("An OVA."));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_episode_tvdb_never_placed_is_left_out() {
|
||||
assert_eq!(mapping(Some(2)).episodes.len(), 2);
|
||||
}
|
||||
|
||||
/// A film or a special as an entry has no place among the series' seasons.
|
||||
#[test]
|
||||
fn an_entry_outside_the_series_numbers_nothing_by_entry() {
|
||||
assert!(
|
||||
mapping(None)
|
||||
.episodes
|
||||
.iter()
|
||||
.all(|episode| episode.entry_season.is_none() && episode.entry_number.is_none())
|
||||
);
|
||||
}
|
||||
|
||||
/// The 16:9 picture, not the banner — the banner is the 4.75:1 strip that
|
||||
/// looked wrong on every desktop tile.
|
||||
#[test]
|
||||
fn the_fanart_is_taken_and_the_banner_is_not() {
|
||||
assert_eq!(mapping(None).fanart.as_deref(), Some("fanart.jpg"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_entry_ani_zip_has_nothing_for_parses_to_nothing() {
|
||||
let mapping = serde_json::from_str::<Payload>("{}")
|
||||
.expect("parse")
|
||||
.into_mapping("en", Some(1));
|
||||
assert!(mapping.episodes.is_empty());
|
||||
assert_eq!(mapping.fanart, None);
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,8 @@
|
||||
//! Title ──▶ Query ──▶ Provider::search ──▶ [Candidate] ──▶ matching::best
|
||||
//! (anilist | tmdb) │
|
||||
//! ▼
|
||||
//! TitleMetadata
|
||||
//! Enrichment ◀── Provider::enrich ◀──────────────────── TitleMetadata
|
||||
//! (episodes, fanart) (anilist relations + ani.zip | tmdb)
|
||||
//! ```
|
||||
//!
|
||||
//! Three things are worth knowing before touching any of it:
|
||||
@@ -28,6 +29,7 @@
|
||||
//! without either depending on this crate.
|
||||
|
||||
pub mod anilist;
|
||||
pub mod anizip;
|
||||
pub mod error;
|
||||
pub mod limiter;
|
||||
pub mod matching;
|
||||
|
||||
@@ -1,35 +1,24 @@
|
||||
//! What every metadata source has to be able to do.
|
||||
|
||||
use pstr_core::metadata::{EpisodeMetadata, ProviderId, TitleMetadata};
|
||||
use pstr_core::library::Title;
|
||||
use pstr_core::metadata::{Enrichment, ProviderId, TitleMetadata};
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::matching::{Candidate, Query};
|
||||
|
||||
/// A source of titles.
|
||||
///
|
||||
/// Deliberately narrow: search, and the episode list of something search
|
||||
/// already matched. Both are requests to a third party, and neither happens
|
||||
/// until the viewer has turned enrichment on — but note what the second one
|
||||
/// does *not* need: it takes the id search returned, so it sends the provider
|
||||
/// nothing about the library it did not already learn from the match.
|
||||
/// Deliberately narrow: search, and enriching something search already
|
||||
/// matched. Both are requests to a third party, and neither happens until the
|
||||
/// viewer has turned enrichment on — but note what the second one does *not*
|
||||
/// need: it takes the id search returned, so it sends nothing about the library
|
||||
/// it did not already learn from the match.
|
||||
///
|
||||
/// Not `#[async_trait]`: this is only ever used through a concrete type or an
|
||||
/// enum, never as `dyn Provider`, so the native `async fn` costs nothing here.
|
||||
pub trait Provider: Send + Sync {
|
||||
fn id(&self) -> ProviderId;
|
||||
|
||||
/// Whether a sequel is a *separate entry* rather than a second season of
|
||||
/// the same one.
|
||||
///
|
||||
/// AniList works that way — `Oshi no Ko`, `Oshi no Ko 2nd Season` and
|
||||
/// `Oshi no Ko 3rd Season` are three ids, each numbering its episodes from
|
||||
/// one — so a library that files all three under one title has to search
|
||||
/// again per season or seasons two upwards get no episode names at all (and,
|
||||
/// worse, would be answered with season one's). TMDB files them as seasons
|
||||
/// of one show and its episode list already carries season numbers, so
|
||||
/// searching per season there would only find the wrong entry.
|
||||
fn seasons_are_separate_entries(&self) -> bool;
|
||||
|
||||
/// Everything the provider thinks `query` might be, in its own order.
|
||||
///
|
||||
/// Scoring is not the provider's job — it returns candidates and
|
||||
@@ -40,13 +29,16 @@ pub trait Provider: Send + Sync {
|
||||
query: &Query,
|
||||
) -> impl std::future::Future<Output = Result<Vec<Candidate>>> + Send;
|
||||
|
||||
/// Every episode the provider lists for a title it already matched.
|
||||
/// The episodes, and any art the search answer lacked, of `found` — the
|
||||
/// entry `title` matched.
|
||||
///
|
||||
/// An empty list is a real answer — a film has no episodes, and plenty of
|
||||
/// series entries have none listed — and the caller caches it rather than
|
||||
/// asking again on the next render.
|
||||
fn episodes(
|
||||
/// `title` is there for its files: a provider that files each sequel as a
|
||||
/// separate entry (AniList) has to know which seasons the library holds to
|
||||
/// know how far along the franchise to look. An empty episode list is a
|
||||
/// real answer — a film has none — and the caller records it.
|
||||
fn enrich(
|
||||
&self,
|
||||
title: &TitleMetadata,
|
||||
) -> impl std::future::Future<Output = Result<Vec<EpisodeMetadata>>> + Send;
|
||||
title: &Title,
|
||||
found: &TitleMetadata,
|
||||
) -> impl std::future::Future<Output = Result<Enrichment>> + Send;
|
||||
}
|
||||
+203
-168
@@ -20,11 +20,12 @@
|
||||
//! Nothing here is requested unless [`MetadataConfig::enabled`] is set. See the
|
||||
//! note in `pstr_core::metadata` on why that is off by default.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use pstr_core::library::{Title, TitleKind, title_key};
|
||||
use pstr_core::metadata::{
|
||||
EpisodeMetadata, MetadataConfig, MetadataRecord, ProviderId, TitleMetadata,
|
||||
Enrichment, MATCH_TTL_SECS, MetadataConfig, MetadataRecord, ProviderId, TitleMetadata,
|
||||
};
|
||||
|
||||
use crate::anilist::AniList;
|
||||
@@ -63,17 +64,10 @@ impl Source {
|
||||
}
|
||||
}
|
||||
|
||||
async fn episodes(&self, title: &TitleMetadata) -> Result<Vec<EpisodeMetadata>> {
|
||||
async fn enrich(&self, title: &Title, found: &TitleMetadata) -> Result<Enrichment> {
|
||||
match self {
|
||||
Self::AniList(provider) => provider.episodes(title).await,
|
||||
Self::Tmdb(provider) => provider.episodes(title).await,
|
||||
}
|
||||
}
|
||||
|
||||
fn seasons_are_separate_entries(&self) -> bool {
|
||||
match self {
|
||||
Self::AniList(provider) => provider.seasons_are_separate_entries(),
|
||||
Self::Tmdb(provider) => provider.seasons_are_separate_entries(),
|
||||
Self::AniList(provider) => provider.enrich(title, found).await,
|
||||
Self::Tmdb(provider) => provider.enrich(title, found).await,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -101,7 +95,9 @@ impl MetadataService {
|
||||
.map_err(Error::Network)?;
|
||||
|
||||
let source = match config.provider {
|
||||
ProviderId::AniList => Source::AniList(AniList::new(http.clone())),
|
||||
ProviderId::AniList => {
|
||||
Source::AniList(AniList::new(http.clone(), config.language.clone()))
|
||||
}
|
||||
ProviderId::Tmdb => Source::Tmdb(Tmdb::new(
|
||||
http.clone(),
|
||||
api_key.ok_or(Error::MissingApiKey(ProviderId::Tmdb))?,
|
||||
@@ -204,136 +200,29 @@ impl MetadataService {
|
||||
}
|
||||
}
|
||||
|
||||
/// What the provider lists as the episodes of a title it matched.
|
||||
/// The episodes of a matched title, and any art its match lacked.
|
||||
///
|
||||
/// Only ever called for a title that already has a match, because the
|
||||
/// provider's own id is what it takes — there is no second search here, and
|
||||
/// nothing about the library goes out that the match did not already send.
|
||||
/// An empty list is a real answer and belongs in the catalog: a film has no
|
||||
/// episodes, and asking again per render is the trap `MetadataRecord`
|
||||
/// exists to avoid.
|
||||
pub async fn episodes(&self, title: &TitleMetadata) -> Result<Vec<EpisodeMetadata>> {
|
||||
let episodes = self.source.episodes(title).await?;
|
||||
///
|
||||
/// An `Err` is a failure to ask and must not be stored: the title stays
|
||||
/// un-enriched and the next match run asks again. An empty answer is an
|
||||
/// answer, and is — see `Catalog::set_enrichment`.
|
||||
pub async fn enrich(&self, title: &Title, found: &TitleMetadata) -> Result<Enrichment> {
|
||||
let enrichment = self.source.enrich(title, found).await?;
|
||||
tracing::debug!(
|
||||
"{}: {} episodes from {}",
|
||||
"{}: {} episodes from {}{}",
|
||||
title.name,
|
||||
episodes.len(),
|
||||
self.provider().label()
|
||||
enrichment.episodes.len(),
|
||||
self.provider().label(),
|
||||
if enrichment.backdrop_url.is_some() {
|
||||
", with a backdrop"
|
||||
} else {
|
||||
""
|
||||
}
|
||||
);
|
||||
Ok(episodes)
|
||||
}
|
||||
|
||||
/// The episode list for a whole title, with a failure treated as an empty
|
||||
/// one.
|
||||
///
|
||||
/// Deliberately not fatal to the title it belongs to: a poster and a
|
||||
/// synopsis that arrived are worth keeping even when the episode request
|
||||
/// was the one that hit the rate limit. The empty result is not cached as
|
||||
/// an answer — the next match run asks again — because the only thing that
|
||||
/// triggers one is a viewer pressing the button.
|
||||
///
|
||||
/// The match itself only ever answers for *one* entry, and on a provider
|
||||
/// that files each sequel separately — AniList — that entry is season one.
|
||||
/// So each further season of the title is searched for by name and its
|
||||
/// episodes are tagged with the season they came from; without that,
|
||||
/// seasons two and three have no episode names at all. See
|
||||
/// [`MetadataService::season_episodes`].
|
||||
pub async fn title_episodes(
|
||||
&self,
|
||||
title: &Title,
|
||||
found: &TitleMetadata,
|
||||
) -> Vec<EpisodeMetadata> {
|
||||
let mut episodes = self
|
||||
.episodes(found)
|
||||
.await
|
||||
.inspect_err(|error| tracing::warn!("episodes for {}: {error}", found.name))
|
||||
.unwrap_or_default();
|
||||
|
||||
if !self.splits_seasons() {
|
||||
return episodes;
|
||||
}
|
||||
|
||||
// Season one is what the title's own match already answered for.
|
||||
let later: Vec<u32> = title
|
||||
.seasons
|
||||
.iter()
|
||||
.filter_map(|season| season.number)
|
||||
.filter(|number| *number > 1)
|
||||
.collect();
|
||||
for season in later {
|
||||
match self.season_episodes(title, season).await {
|
||||
Ok(found) => episodes.extend(found),
|
||||
Err(error) => {
|
||||
tracing::warn!("episodes for {} season {season}: {error}", title.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
episodes
|
||||
}
|
||||
|
||||
/// Whether this provider files a sequel as its own entry, so that a title
|
||||
/// with several seasons has to be searched for once per season.
|
||||
pub fn splits_seasons(&self) -> bool {
|
||||
self.source.seasons_are_separate_entries()
|
||||
}
|
||||
|
||||
/// The episodes of one season of `title`, for a provider that files each
|
||||
/// season separately.
|
||||
///
|
||||
/// A second search, not a second guess: the title's own match is season
|
||||
/// one's entry, and asking it about episode one of season two would answer
|
||||
/// with episode one of season one. So the season is searched for by the
|
||||
/// name the provider itself uses — `Oshi no Ko 2nd Season`, and `Oshi no Ko
|
||||
/// Season 2` when that finds nothing, which are the two conventions between
|
||||
/// them covering everything AniList carries.
|
||||
///
|
||||
/// A miss is an empty list rather than an error: a season the provider has
|
||||
/// never heard of is ordinary, and it leaves those rows named by their
|
||||
/// filenames exactly as they were before.
|
||||
pub async fn season_episodes(
|
||||
&self,
|
||||
title: &Title,
|
||||
season: u32,
|
||||
) -> Result<Vec<EpisodeMetadata>> {
|
||||
if !self.splits_seasons() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
let mut found = None;
|
||||
for name in season_names(&title.name, season) {
|
||||
// No year: a sequel airs years after the series the library named
|
||||
// the folder for, so carrying the title's year over would penalise
|
||||
// the very entry being looked for.
|
||||
let query = Query::new(name, None, title.kind);
|
||||
let candidates = self.source.search(&query).await?;
|
||||
if let Some(matched) = matching::best(&query, candidates) {
|
||||
found = Some(matched);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let Some(found) = found else {
|
||||
tracing::debug!("{}: no entry for season {season}", title.name);
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
|
||||
let episodes = self.source.episodes(&found).await?;
|
||||
tracing::debug!(
|
||||
"{} season {season}: {} episodes from {:?}",
|
||||
title.name,
|
||||
episodes.len(),
|
||||
found.name
|
||||
);
|
||||
// The entry numbers its own episodes from one; what makes them season
|
||||
// two's is which entry they came from, and that has to be recorded here
|
||||
// or nothing can look them up again.
|
||||
Ok(episodes
|
||||
.into_iter()
|
||||
.map(|episode| EpisodeMetadata {
|
||||
season: Some(season),
|
||||
..episode
|
||||
})
|
||||
.collect())
|
||||
Ok(enrichment)
|
||||
}
|
||||
|
||||
/// Download one piece of artwork.
|
||||
@@ -351,29 +240,107 @@ impl MetadataService {
|
||||
}
|
||||
}
|
||||
|
||||
/// What to search for when looking for one season of a series, best first.
|
||||
///
|
||||
/// `2nd Season` is AniList's own convention and matches its romaji titles
|
||||
/// outright; `Season 2` is what its English titles and most western libraries
|
||||
/// use. Season one is never searched for this way — that is the title's own
|
||||
/// match.
|
||||
fn season_names(title: &str, season: u32) -> Vec<String> {
|
||||
vec![
|
||||
format!("{title} {} Season", ordinal(season)),
|
||||
format!("{title} Season {season}"),
|
||||
]
|
||||
/// What one match run has to do about one title.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum Work {
|
||||
/// Search for it, and enrich it if it matches.
|
||||
Match(Title),
|
||||
/// It is already matched; only its enrichment is missing or old.
|
||||
Enrich(Title, Box<TitleMetadata>),
|
||||
}
|
||||
|
||||
/// `2` → `2nd`. English ordinals, because that is what the provider writes.
|
||||
fn ordinal(number: u32) -> String {
|
||||
let suffix = match (number % 10, number % 100) {
|
||||
(_, 11..=13) => "th",
|
||||
(1, _) => "st",
|
||||
(2, _) => "nd",
|
||||
(3, _) => "rd",
|
||||
_ => "th",
|
||||
impl Work {
|
||||
pub fn title(&self) -> &Title {
|
||||
match self {
|
||||
Self::Match(title) | Self::Enrich(title, _) => title,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What a piece of [`Work`] found, ready to store.
|
||||
#[derive(Debug)]
|
||||
pub struct Outcome {
|
||||
pub title_key: String,
|
||||
/// The new answer, for [`Work::Match`]. Store it — misses included.
|
||||
pub record: Option<MetadataRecord>,
|
||||
/// What enriching the match found, when it was asked and answered. `None`
|
||||
/// after a failure, which must leave the title un-enriched so the next run
|
||||
/// asks again.
|
||||
pub enrichment: Option<Enrichment>,
|
||||
}
|
||||
|
||||
/// Decide what a match run over `titles` has to do.
|
||||
///
|
||||
/// Two kinds of work, and the difference is a request saved: a title that
|
||||
/// already has a good match but has not been enriched — or was enriched longer
|
||||
/// ago than a match is trusted — needs enriching only, and searching for it
|
||||
/// again would ask the provider something it has already answered.
|
||||
///
|
||||
/// A hand-picked entry is never searched for again, not even when `force`d —
|
||||
/// that means "the automatic answers are wrong", and re-deciding the one title
|
||||
/// the viewer already fixed by hand is the opposite of what they asked for. It
|
||||
/// is still enriched.
|
||||
pub fn plan(
|
||||
titles: Vec<Title>,
|
||||
stored: &HashMap<String, MetadataRecord>,
|
||||
enriched: &HashMap<String, (ProviderId, i64)>,
|
||||
provider: ProviderId,
|
||||
force: bool,
|
||||
) -> Vec<Work> {
|
||||
let now = now();
|
||||
let mut work = Vec::new();
|
||||
for title in titles {
|
||||
let record = stored.get(&title.key);
|
||||
let pinned = record.is_some_and(|record| record.manual && record.provider == provider);
|
||||
if !pinned && (force || !is_usable(record, provider)) {
|
||||
work.push(Work::Match(title));
|
||||
continue;
|
||||
}
|
||||
let fresh = enriched.get(&title.key).is_some_and(|(asked, at)| {
|
||||
*asked == provider && now.saturating_sub(*at) < MATCH_TTL_SECS
|
||||
});
|
||||
if let Some(found) = record.and_then(|record| record.metadata.clone())
|
||||
&& !fresh
|
||||
{
|
||||
work.push(Work::Enrich(title, Box::new(found)));
|
||||
}
|
||||
}
|
||||
work
|
||||
}
|
||||
|
||||
impl MetadataService {
|
||||
/// Do one piece of [`Work`].
|
||||
///
|
||||
/// A failed search is an `Err` and nothing is stored. A failed enrichment
|
||||
/// after a successful search is not: the match is worth keeping, and the
|
||||
/// title is simply left un-enriched — a poster and a synopsis that arrived
|
||||
/// are worth showing even when the episode request was the one that hit the
|
||||
/// rate limit.
|
||||
pub async fn work(&self, work: &Work) -> Result<Outcome> {
|
||||
match work {
|
||||
Work::Match(title) => {
|
||||
let record = self.record(title).await?;
|
||||
let enrichment = match &record.metadata {
|
||||
Some(found) => self
|
||||
.enrich(title, found)
|
||||
.await
|
||||
.inspect_err(|error| tracing::warn!("enrich {}: {error}", title.name))
|
||||
.ok(),
|
||||
None => None,
|
||||
};
|
||||
format!("{number}{suffix}")
|
||||
Ok(Outcome {
|
||||
title_key: record.title_key.clone(),
|
||||
record: Some(record),
|
||||
enrichment,
|
||||
})
|
||||
}
|
||||
Work::Enrich(title, found) => Ok(Outcome {
|
||||
title_key: title.key.clone(),
|
||||
record: None,
|
||||
enrichment: Some(self.enrich(title, found).await?),
|
||||
}),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a stored answer can be used as-is.
|
||||
@@ -412,22 +379,90 @@ mod tests {
|
||||
assert!(MetadataService::new(&tmdb, Some("key".into())).is_ok());
|
||||
}
|
||||
|
||||
/// The names a season is searched for by, in the order they are tried.
|
||||
fn title(key: &str) -> Title {
|
||||
Title {
|
||||
key: key.into(),
|
||||
name: key.into(),
|
||||
year: None,
|
||||
kind: TitleKind::Series,
|
||||
seasons: Vec::new(),
|
||||
share_ids: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn matched(key: &str, manual: bool) -> MetadataRecord {
|
||||
MetadataRecord {
|
||||
title_key: key.into(),
|
||||
provider: ProviderId::AniList,
|
||||
metadata: Some(TitleMetadata {
|
||||
provider: ProviderId::AniList,
|
||||
remote_id: "1".into(),
|
||||
name: key.into(),
|
||||
original_name: None,
|
||||
overview: None,
|
||||
year: None,
|
||||
kind: TitleKind::Series,
|
||||
poster_url: None,
|
||||
backdrop_url: None,
|
||||
rating: None,
|
||||
genres: Vec::new(),
|
||||
episodes: None,
|
||||
url: None,
|
||||
}),
|
||||
fetched_at: now(),
|
||||
manual,
|
||||
}
|
||||
}
|
||||
|
||||
fn kinds(work: &[Work]) -> Vec<(&str, &str)> {
|
||||
work.iter()
|
||||
.map(|work| match work {
|
||||
Work::Match(title) => (title.key.as_str(), "match"),
|
||||
Work::Enrich(title, _) => (title.key.as_str(), "enrich"),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// A matched title is enriched once and then left alone until its
|
||||
/// enrichment is as old as a match is trusted; an unasked title is matched.
|
||||
#[test]
|
||||
fn a_season_is_searched_for_the_way_the_provider_names_it() {
|
||||
fn a_run_matches_the_unasked_and_enriches_the_unenriched() {
|
||||
let stored = HashMap::from([
|
||||
("enriched".to_string(), matched("enriched", false)),
|
||||
("stale".to_string(), matched("stale", false)),
|
||||
("bare".to_string(), matched("bare", false)),
|
||||
]);
|
||||
let enriched = HashMap::from([
|
||||
("enriched".to_string(), (ProviderId::AniList, now())),
|
||||
(
|
||||
"stale".to_string(),
|
||||
(ProviderId::AniList, now() - MATCH_TTL_SECS - 1),
|
||||
),
|
||||
]);
|
||||
let titles = ["enriched", "stale", "bare", "new"].map(title).to_vec();
|
||||
|
||||
let work = plan(titles, &stored, &enriched, ProviderId::AniList, false);
|
||||
assert_eq!(
|
||||
season_names("Oshi no Ko", 2),
|
||||
vec!["Oshi no Ko 2nd Season", "Oshi no Ko Season 2"]
|
||||
kinds(&work),
|
||||
vec![("stale", "enrich"), ("bare", "enrich"), ("new", "match")]
|
||||
);
|
||||
}
|
||||
|
||||
/// "Match again" re-searches everything except what the viewer pinned —
|
||||
/// and a pinned title that was never enriched is still enriched.
|
||||
#[test]
|
||||
fn a_forced_run_leaves_a_pinned_match_alone_but_still_enriches_it() {
|
||||
let stored = HashMap::from([
|
||||
("pinned".to_string(), matched("pinned", true)),
|
||||
("automatic".to_string(), matched("automatic", false)),
|
||||
]);
|
||||
let titles = ["pinned", "automatic"].map(title).to_vec();
|
||||
|
||||
let work = plan(titles, &stored, &HashMap::new(), ProviderId::AniList, true);
|
||||
assert_eq!(
|
||||
kinds(&work),
|
||||
vec![("pinned", "enrich"), ("automatic", "match")]
|
||||
);
|
||||
assert_eq!(ordinal(1), "1st");
|
||||
assert_eq!(ordinal(3), "3rd");
|
||||
assert_eq!(ordinal(4), "4th");
|
||||
// The teens are the ones a naive rule gets wrong, and a show does reach
|
||||
// them: `11th` is not `11st`.
|
||||
assert_eq!(ordinal(11), "11th");
|
||||
assert_eq!(ordinal(12), "12th");
|
||||
assert_eq!(ordinal(13), "13th");
|
||||
assert_eq!(ordinal(21), "21st");
|
||||
}
|
||||
|
||||
/// A record from the other provider is never usable — switching provider has
|
||||
|
||||
@@ -10,8 +10,8 @@
|
||||
//! better evidence than the guess, and [`crate::matching`] only nudges on kind
|
||||
//! anyway.
|
||||
|
||||
use pstr_core::library::TitleKind;
|
||||
use pstr_core::metadata::{EpisodeMetadata, ProviderId, TitleMetadata};
|
||||
use pstr_core::library::{Title, TitleKind};
|
||||
use pstr_core::metadata::{Enrichment, EpisodeMetadata, ProviderId, TitleMetadata};
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::error::{Error, Result};
|
||||
@@ -73,12 +73,6 @@ impl Provider for Tmdb {
|
||||
ProviderId::Tmdb
|
||||
}
|
||||
|
||||
/// One show, every season under it — and [`Self::episodes`] already returns
|
||||
/// them with their season numbers.
|
||||
fn seasons_are_separate_entries(&self) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
async fn search(&self, query: &Query) -> Result<Vec<Candidate>> {
|
||||
if self.api_key.trim().is_empty() {
|
||||
return Err(Error::MissingApiKey(ProviderId::Tmdb));
|
||||
@@ -101,6 +95,18 @@ impl Provider for Tmdb {
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// The episode list, and nothing else: the match already carried a proper
|
||||
/// backdrop, and TMDB files every season under one show, so the library's
|
||||
/// seasons need no walking.
|
||||
async fn enrich(&self, _title: &Title, found: &TitleMetadata) -> Result<Enrichment> {
|
||||
Ok(Enrichment {
|
||||
episodes: self.episodes(found).await?,
|
||||
backdrop_url: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
impl Tmdb {
|
||||
/// Two requests: the show, to learn which seasons exist, and then those
|
||||
/// seasons appended onto one more.
|
||||
///
|
||||
@@ -153,6 +159,9 @@ impl Provider for Tmdb {
|
||||
EpisodeMetadata {
|
||||
season: Some(number),
|
||||
number: episode.episode_number,
|
||||
absolute: None,
|
||||
entry_season: None,
|
||||
entry_number: None,
|
||||
name: episode.name.clone().filter(|name| !name.is_empty()),
|
||||
overview: episode.overview.clone().filter(|text| !text.is_empty()),
|
||||
still_url: episode
|
||||
@@ -165,9 +174,7 @@ impl Provider for Tmdb {
|
||||
}
|
||||
Ok(episodes)
|
||||
}
|
||||
}
|
||||
|
||||
impl Tmdb {
|
||||
/// One authenticated GET, paced, with TMDB's failures named.
|
||||
///
|
||||
/// TMDB is far more generous than AniList and a library-sized run rarely
|
||||
|
||||
@@ -146,6 +146,51 @@ and [B44](#b44--r8-broke-the-app-under-instrumentation-in-five-places).
|
||||
|
||||
## Fixed
|
||||
|
||||
### B61 — later seasons were captioned with season one's episode names
|
||||
|
||||
**Symptom.** On a 42-title AniList library, Attack on Titan seasons 2–4 and
|
||||
Jigokuraku season 2 showed season one's episode names. Dan Da Dan, Kaiju No. 8
|
||||
and The Apothecary Diaries had a handful of names numbered past the end of the
|
||||
season. Bleach's had 20 rows called `Untitled`. Bleach `S17` (the Thousand-Year
|
||||
Blood War) and Mushishi `S02`/`S03` had none at all. Jujutsu Kaisen 0 kept the
|
||||
TV series' names after it was re-pinned to the film. Every desktop tile drew a
|
||||
blurred strip of AniList banner.
|
||||
|
||||
**Cause.** Four, independent.
|
||||
|
||||
- `season_episodes` searched for `<title> 2nd Season`, and the scorer, seeing
|
||||
the base entry's alias inside the query, picked the base entry again. That
|
||||
search was [B11](#b11--seasons-past-the-first-had-no-episode-names-and-were-one-query-from-having-the-wrong-ones)'s
|
||||
fix, and it only worked where the sequel's name scored higher than the base's.
|
||||
- `streamingEpisodes` is what a streaming site published: partial, and
|
||||
numbered however that site counted.
|
||||
- A release's season is not an AniList entry. `S17` is four entries, and
|
||||
Mushoku Tensei's `S01` is two.
|
||||
- An empty episode list was never written, so a re-match kept the old one.
|
||||
`bannerImage`, a 4.75:1 strip, was stored as the 16:9 backdrop.
|
||||
|
||||
**Fix.** AniList walks `SEQUEL` relations from the match. Each entry's episodes
|
||||
come from ani.zip with TVDB's season, number and absolute number, plus the
|
||||
entry's own numbering. `EpisodeGuide::get` meets whichever of those a filename
|
||||
used without ever crossing seasons. Enrichment is stored as one unit,
|
||||
replaced even when empty and recorded as asked. A changed match drops the old
|
||||
entry's rows at once. The backdrop is ani.zip's 1920×1080 fanart, and the
|
||||
banner is no longer stored. Schema v9 deletes the bad rows. See
|
||||
[METADATA.md](METADATA.md).
|
||||
|
||||
**Verified.** `EpisodeGuide` tests in `pstr_core::metadata` (absolute numbers
|
||||
in a season folder, by-entry seasons, no season crossing), and the catalog's
|
||||
`an_empty_enrichment_clears_the_episodes_the_last_match_left`,
|
||||
`pinning_a_different_entry_drops_the_old_entrys_episodes_at_once` and
|
||||
`upgrading_drops_anilists_episode_rows_and_banners_but_keeps_its_matches`.
|
||||
Live, against a copy of the same library through `pstr metadata match`:
|
||||
|
||||
- 0 lookups failed.
|
||||
- Every numbered file is answered except 2 Attack on Titan files, 2 Bleach
|
||||
files and one Mushoku Tensei special, which TVDB does not list.
|
||||
- 29 of 44 matched titles have fanart, and none has a banner.
|
||||
- A second run asks nothing.
|
||||
|
||||
### B60 — a failed "Add and crawl" left the crawl spinner running forever
|
||||
|
||||
**Symptom.** Pasting a link without its `#` fragment, or one already added,
|
||||
|
||||
+20
-24
@@ -106,26 +106,22 @@ card (`App::tick_up_next`, `ui::player::up_next_card`) with *Play now* and
|
||||
chapters, autoplay on, a next episode, and playback actually running — a paused
|
||||
episode is one somebody walked away from.
|
||||
|
||||
Done: **per-episode metadata.** `Provider::episodes` fetches an episode list for
|
||||
a title that already matched (AniList `streamingEpisodes`, which is titles and
|
||||
thumbnails only and no synopsis; TMDB `/tv/{id}` plus every season appended onto
|
||||
one more request), stored in `episode_metadata` — schema **v4**. The reconciling
|
||||
between a release's numbering and a provider's lives in one tested place,
|
||||
`EpisodeGuide::get`: the season and number the filename states, then the
|
||||
provider's absolute numbering, then season one for a file that states no season
|
||||
at all. That last fallback is what makes `#057` line up with TMDB filing a
|
||||
64-episode anime under one season. Episode names show on the title page's rows,
|
||||
in the player's title strip and in the "next:" line autoplay prints.
|
||||
Done: **per-episode metadata.** `Provider::enrich` fetches the episode list —
|
||||
and, for AniList, the series' fanart — of a title that already matched, and
|
||||
`Catalog::set_enrichment` stores both and records that it asked (schema **v9**),
|
||||
so an empty answer is not re-asked every run. TMDB is `/tv/{id}` plus every
|
||||
season appended onto one more request. AniList walks the match's `SEQUEL`
|
||||
relations and takes each entry's episodes from ani.zip, which carries TVDB's
|
||||
season, number and absolute number — the numbering release groups use. The
|
||||
reconciling between a release's numbering and a provider's lives in one tested
|
||||
place, `EpisodeGuide::get`. [METADATA.md](METADATA.md) has the audit that drove
|
||||
this, the order `get` tries things in, and what comes next.
|
||||
|
||||
**A season is a separate lookup on AniList.** `Provider::seasons_are_separate_entries`
|
||||
is what splits the two providers: TMDB files every season under one show and
|
||||
returns them numbered, while AniList makes each sequel its own entry numbering
|
||||
from one (`[Oshi no Ko] 2nd Season`, id 166531, is not `[Oshi no Ko]`, id
|
||||
150672). So for AniList each season past the first is searched for by name —
|
||||
`<title> 2nd Season`, then `<title> Season 2` — and its episodes are stored
|
||||
tagged with that season. The absolute fallback in `EpisodeGuide::get` stops at
|
||||
season one for the same reason: without that it answered `S02E01` with season
|
||||
one's episode one, which is not a missing caption but a wrong one.
|
||||
The first version searched AniList for `<title> 2nd Season` per season and took
|
||||
episode names from `streamingEpisodes`; both were wrong often enough to show —
|
||||
the search found the base entry again and captioned Attack on Titan's later
|
||||
seasons with season one's names, and `streamingEpisodes` is whatever a streaming
|
||||
site published. v9 deletes everything they stored.
|
||||
|
||||
Done: **matching a title by hand.** `MATCH_FLOOR` is set where a wrong poster
|
||||
costs more than none, and the price of that is a few titles it refuses to decide
|
||||
@@ -145,11 +141,11 @@ rather than storing a miss, so the next run treats the title as one it has never
|
||||
asked about. Switching provider still clears everything, hand-picked rows
|
||||
included: a `remote_id` means nothing to the other provider.
|
||||
|
||||
Worth knowing before chasing a missing name: AniList's episode titles come from
|
||||
`streamingEpisodes`, which is what streaming sites published, and for some shows
|
||||
it is simply **empty** — all three Oshi no Ko entries have none, while Fullmetal
|
||||
Alchemist Brotherhood has 64. There is nothing to fix on our side for those;
|
||||
TMDB is the provider that has them.
|
||||
Worth knowing before chasing a missing name: ani.zip only lists what TVDB has
|
||||
placed. A file it cannot answer is usually a special (`S02E00`), an episode TVDB
|
||||
skipped, or one too new to be listed — Attack on Titan's last two files and two
|
||||
of Bleach's forty-four are the standing examples. `pstr metadata match` prints
|
||||
how many episodes each title got.
|
||||
|
||||
Note the glyph trap: egui's bundled fonts cover the transport icons
|
||||
(`⏮ ⏭ ⏸ ▶ 🔊 🔇 ⛶`) and *not* `← 💬 ▣`, which draw as an empty box. Anything new
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
# Metadata
|
||||
|
||||
How a library of filenames becomes posters, synopses and episode names, what
|
||||
was wrong with the first version of it, and where it is going. The privacy
|
||||
stance from `pstr_core::metadata` holds throughout: nothing here is requested
|
||||
until the viewer turns enrichment on, and every new source named below sits
|
||||
behind that same switch.
|
||||
|
||||
## What a real library showed
|
||||
|
||||
Audited against a 42-title, 758-file anime library matched on AniList
|
||||
(schema v8, October 2026). The match itself — title to AniList entry — was
|
||||
good. What hung off it was not:
|
||||
|
||||
1. **Later seasons captioned with season one's names.** Attack on Titan
|
||||
seasons 2–4 and Jigokuraku season 2 all stored season one's episode list.
|
||||
The per-season lookup searched for `Attack on Titan 2nd Season`, and the
|
||||
similarity scorer, seeing the base entry's alias inside the query, picked the
|
||||
base entry again. That is the failure `EpisodeGuide::get` already warned
|
||||
about: not a missing caption but a wrong one.
|
||||
2. **`streamingEpisodes` is not an episode list.** It is what a streaming site
|
||||
published, partial and numbered however that site numbered it. Dan Da Dan
|
||||
had 5 rows numbered 13–17, Kaiju No. 8 had 4–15, The Apothecary Diaries had
|
||||
25–48 under two seasons at once, Bleach had 20 rows called `Untitled`, Akira
|
||||
had a trailer. No row anywhere had a synopsis — AniList keeps none.
|
||||
3. **A release's season is not an AniList entry.** Bleach `S17` is the
|
||||
Thousand-Year Blood War, four separate AniList entries. Mushoku Tensei `S01`
|
||||
is two (11 + 12 episodes). Attack on Titan's season folders hold
|
||||
absolutely-numbered files — `S2` is episodes 26–37. Mushishi's `S03` is
|
||||
*Zoku Shou 2*, which TVDB files as the second half of season two.
|
||||
4. **Answers outlived their match.** `jujutsu kaisen 0 movie` kept the TV
|
||||
series' 24 episode names after it was re-pinned to the film: an empty episode
|
||||
list was never written, so the old one stayed.
|
||||
5. **Desktop art was the wrong shape.** The 16:9 tile and the library's hero
|
||||
prefer `backdrop_url`, which on AniList is `bannerImage` — a 1900×400 strip,
|
||||
4.75:1, cached at 640×135. Cropped to 16:9 and stretched, that is a blurred
|
||||
sliver from the middle of a banner. Android looks right because it draws the
|
||||
2:3 cover.
|
||||
6. **"Newest" sorts on nothing.** It reads the year from the filename, which 2
|
||||
of 42 titles state.
|
||||
|
||||
## Sources
|
||||
|
||||
| Source | Key | What it is good for |
|
||||
|---|---|---|
|
||||
| **AniList** GraphQL | none | Identity. Search, every alias, cover art, relations (`SEQUEL`, `PREQUEL`, `SIDE_STORY`…), studios, staff, tags, format, season, popularity, next airing episode, the MAL id. |
|
||||
| **ani.zip** `api.ani.zip/mappings?anilist_id=` | none | Everything AniList lacks, per AniList id: TVDB season, episode and absolute numbers; episode titles in several languages; TVDB synopses; 16:9 stills; air dates; runtime. Series art: TVDB **fanart at 1920×1080**, poster, banner, logo. Cross-ids for MAL, AniDB, TVDB, TMDB, Kitsu, IMDb. A community service — cache what it says. |
|
||||
| **TMDB** | free key | Film backdrops and logos; live-action TV. Already implemented. |
|
||||
| **TVmaze** | none | Live-action TV episodes, keyless. |
|
||||
| **AniSkip** | none | Opening and ending timestamps by MAL id and episode. |
|
||||
| **Jikan** (MAL) | none | Filler and recap flags. Timed out when tried; optional. |
|
||||
| **manami-project/anime-offline-database** | none | Every anime with its synonyms, ids and relations, as one 6 MB `.zst`. Matching against it locally sends no title anywhere. |
|
||||
| AniDB, TVDB directly | registration / paid | Covered by ani.zip, which republishes their data. |
|
||||
|
||||
The fact the whole design turns on: **release groups number files the way TVDB
|
||||
does.** `Bleach - S17E01` is TVDB season 17; Attack on Titan's 26–96 are TVDB
|
||||
absolute numbers. ani.zip gives both for every AniList entry, so the join
|
||||
between a filename and an episode is a lookup, not a guess.
|
||||
|
||||
## Phase 1 — correct data (done)
|
||||
|
||||
**Seasons by relation, not by name.** `AniList::enrich` starts at the matched
|
||||
entry and follows its `SEQUEL` edges — TV formats first, a special or film only
|
||||
when it is the sole link onward (Mushishi's chain runs through an OVA). Each
|
||||
entry's episodes come from ani.zip. The walk stops as soon as every numbered file
|
||||
in the title has an answer, at the end of the chain, or after
|
||||
`MAX_CHAIN_ENTRIES`, so a one-season show costs one ani.zip request and no
|
||||
relations query at all.
|
||||
|
||||
**One episode, three numberings.** `EpisodeMetadata` carries TVDB's
|
||||
`(season, number)`, TVDB's `absolute` number, and its `(entry_season,
|
||||
entry_number)` — the position of its entry among the franchise's TV entries,
|
||||
and its number within that entry, which is how a release that follows AniList
|
||||
rather than TVDB numbers its seasons. `EpisodeGuide::get` tries, in order:
|
||||
|
||||
1. the season and number the file states;
|
||||
2. for a file in no season or season one, the provider's own absolute list,
|
||||
then season one;
|
||||
3. the absolute number — but only into the season the file states, so an
|
||||
absolutely-numbered season folder (`Attack on Titan/S2/… - 26`) finds
|
||||
S2E1, and `S03E13` can never be answered with season one's 13th;
|
||||
4. the entry numbering, only for a season TVDB does not have at all — which is
|
||||
what makes Mushishi `S03E01` *Zoku Shou 2* episode 1 while Mushoku Tensei
|
||||
`S02E01`, a season TVDB does have, stays TVDB's.
|
||||
|
||||
`streamingEpisodes` is gone.
|
||||
|
||||
**Enrichment is recorded, empty or not.** A title's episode list and its
|
||||
fanart are one *enrichment*, stored together by `Catalog::set_enrichment` in one
|
||||
transaction that always replaces the old episode rows — so an empty answer
|
||||
clears what a previous match left behind. `title_metadata.enriched_at` records
|
||||
that it was asked, which is what keeps a film with no episodes from being
|
||||
re-asked on every match run. A new match clears it; a failed enrichment leaves it
|
||||
clear, so the title is asked again.
|
||||
|
||||
**No banner on a 16:9 tile.** AniList's `bannerImage` is no longer stored as a
|
||||
backdrop. The backdrop is ani.zip's fanart where there is one; otherwise the tile
|
||||
letterboxes the poster.
|
||||
|
||||
**Schema v9** adds the three numbering columns and `enriched_at`, deletes every
|
||||
stored AniList episode row and every AniList banner-as-backdrop — all of it
|
||||
`streamingEpisodes` data and wrong-shaped art — and leaves every title
|
||||
un-enriched, so the next match run (which the app starts on its own) refills
|
||||
them. Matches, hand-picked ones included, are kept.
|
||||
|
||||
## Phase 2 — art and order
|
||||
|
||||
- **Art slots** instead of one backdrop: poster (2:3), fanart (16:9), banner
|
||||
(ultra-wide), logo. The hero and the 16:9 tiles take fanart; the banner only
|
||||
ever fills an ultra-wide strip.
|
||||
- **A larger size class for the hero.** `THUMBNAIL_MAX_EDGE` is 640 for every
|
||||
picture; the hero is drawn up to ~1600 px wide.
|
||||
- **A poster grid on desktop**, as an option or the default — AniList's covers
|
||||
are the best art this library has, and Android already proves the layout.
|
||||
- **More of the AniList answer kept**: format, season and year, studios,
|
||||
director, tags, popularity, status, next airing episode — all in the search
|
||||
response already, so no extra request.
|
||||
- **Sorts**: by the displayed name (English / romaji / folder, as a setting),
|
||||
release season, rating, popularity, recently added (needs a `first_seen`
|
||||
column), in-progress first.
|
||||
- **Franchise grouping** from relations: the Evangelion TV series, its two films
|
||||
and the Rebuild folder; Made in Abyss and its three films; Mushishi and its
|
||||
specials; the Heaven's Feel trilogy. One tile each, with the parts in release
|
||||
order on the title page.
|
||||
- **Shelves**: genre, studio, director — this library has a Satoshi Kon shelf
|
||||
waiting in it — format, decade, airing now.
|
||||
|
||||
## Phase 3 — reach
|
||||
|
||||
- **Local matching** against the offline database, so only ids leave the
|
||||
machine.
|
||||
- **Routing by title**: AniList first, then TMDB (with a key) or TVmaze for
|
||||
whatever is not anime — Game of Thrones in a mostly-anime share.
|
||||
- **Folders of films** matched per file: four Rebuild films in one folder are
|
||||
four entries, not one.
|
||||
- **Skip intro** from AniSkip for files without chapters, and filler flags from
|
||||
Jikan.
|
||||
Reference in new issue
Block a user