From f6c505907a0586f59cc01c848b9c5db161de2bc3 Mon Sep 17 00:00:00 2001 From: Nils Pukropp Date: Sun, 1 Nov 2026 13:46:52 +0100 Subject: [PATCH] Add discovery modes, track tags and an artist graph Tracks get tags from their title, description, artist name and BPM (styles such as riddim, versions such as VIP/dub/flip, and tempo), stored in track_tags and shown on cards. Related pages the crawler reads are kept in track_related, which links artists to each other. The feed takes ?mode=: riddim (140 BPM, riddim artists and their graph neighbours first), fresh, deep cuts, and an artist radio that plays an artist and the artists around them. Opening a radio crawls that artist's related pages first. The crawler searches riddim-adjacent terms first. The filters panel picks the feed, each card has a radio button (r), and a chip in the top bar returns to the normal feed. --- README.md | 34 +++- backend/src/crawler.rs | 40 ++++- backend/src/db.rs | 158 ++++++++++++++++- backend/src/http/app.rs | 10 +- backend/src/http/tests.rs | 37 ++++ backend/src/main.rs | 1 + backend/src/ranker.rs | 265 ++++++++++++++++++++++++++++- backend/src/tags.rs | 105 ++++++++++++ backend/templates/datenschutz.html | 2 +- web/index.html | 10 ++ web/src/api.ts | 16 +- web/src/cards.ts | 4 + web/src/main.ts | 67 +++++++- web/src/style.css | 24 ++- 14 files changed, 737 insertions(+), 36 deletions(-) create mode 100644 backend/src/tags.rs diff --git a/README.md b/README.md index 97ba520..d07712f 100644 --- a/README.md +++ b/README.md @@ -120,14 +120,17 @@ visitor address reaches pl8list or a CDN. | swipe up / down | ↓ j · ↑ k | next / previous | | drag waveform | drag waveform · ← → | seek | | follow button | f | follow the artist | +| radio button | r | artist radio (more like them) | | ✕ button | x | not for me (skip, show less) | | share button | s | share / copy link | | | m | mute | | | ? · Esc | help · close panels | Lock-screen, notification and headphone buttons control playback too (Media Session API). -"likes" lists your likes newest first; ▶ plays one right away. Filters (free only, BPM range) -are remembered per browser. The site can be installed to the home screen as an app. +"likes" lists your likes newest first; ▶ plays one right away. The filters panel picks the +feed (for you, riddim, fresh, deep cuts); "radio" on a card starts an artist radio, and the chip +in the top bar goes back to your feed. Feed and filters (free only, BPM range) are remembered +per browser. The site can be installed to the home screen as an app. ## Front end @@ -183,8 +186,31 @@ its artist's catalogue are crawled first. freshness, artist affinity (from their likes, completions, skips, "not for me" and follows), closeness to the BPM and key of tracks they liked, whether it is related to something they liked, and some random exploration. Artists are then spread out so you don't get five tracks from one person in a -row. Weights live in a `Ranker`; `?mode=` on `/local/feed` picks one, so discovery modes -can be added next to `Classic`. +row. Weights live in a `Ranker`, which can also narrow the candidates and add a bonus; +`?mode=` on `/local/feed` picks one: + +| mode | what it plays | +| -------------------------- | ------------------------------------------------------------------------------ | +| (none) / `classic` | the mix above | +| `riddim` | 140 BPM (or 70) tracks, riddim first | +| `fresh` | uploads of the last 45 days, newest weighted up | +| `deep` | tracks with 15 saves or fewer; popularity counts against a track | +| `artist&artist=` | artist radio: the artist and their neighbours in the artist graph (two hops) | + +**Tags** (`backend/src/tags.rs`). pl8list has no genre field, so tags come from words in the +title, description and artist name, and from the BPM: styles (`riddim`, `tearout`, `wonk`, `dnb`, +…), versions (`vip`, `dub`, `dubpack`, `remix`, `flip`, `special`, `collab`, `wip` for v1/v2…) +and tempo (`140`, `170`). They are stored in `track_tags`, sent with each track and shown on the +card. Changing the rules means bumping `tags::VERSION`; the catalogue is retagged on the next +start. + +**Artist graph.** Every related page the crawler reads is stored in `track_related` (which +artists pl8list lists next to a track). Two artists are linked by how often that happens, in +either direction. The artist radio uses the graph directly. Riddim mode uses it to spread the +sparse `riddim` tag: an artist counts as riddim by their own tagged tracks (three count fully) +or by half of what the artists next to them have. Opening a radio crawls that artist and the +related pages of their top tracks first, so a new radio fills in within seconds. The crawler +also searches riddim-adjacent terms first and prefers related pages of riddim tracks. **Audio.** The browser only talks to this server. HLS playlists and segments are proxied from pl8list as a guest, so users hear previews of paid tracks and full free tracks, the same as a diff --git a/backend/src/crawler.rs b/backend/src/crawler.rs index 3b624cd..422d5b5 100644 --- a/backend/src/crawler.rs +++ b/backend/src/crawler.rs @@ -28,16 +28,19 @@ use crate::{ }; /// Search needs at least 3 characters. Genre-ish words find lots of artist handles. +/// Riddim and its neighbours are searched first. +const STYLE_TERMS: &[&str] = &["riddim", "dubstep", "tearout", "dubz", "wonk", "deathstep", "brostep", "halftime", "140"]; const SEARCH_TERMS: &[&str] = &[ - "dub", "bass", "dnb", "jungle", "garage", "grime", "riddim", "dubstep", "trap", "techno", - "house", "beats", "music", "sound", "prod", "audio", "records", "drill", "breaks", "wave", - "deep", "dark", "the", "official", "dj", "mc", "lofi", "halftime", "neuro", "tek", + "dub", "bass", "dnb", "jungle", "garage", "grime", "trap", "techno", "house", "beats", "music", + "sound", "prod", "audio", "records", "drill", "breaks", "wave", "deep", "dark", "the", + "official", "dj", "mc", "lofi", "neuro", "tek", ]; // Lower number = sooner. const P_FEEDBACK: i64 = 0; const P_SEED: i64 = 10; const P_ARTIST_FIRST: i64 = 20; +const P_STYLE: i64 = 15; const P_RELATED: i64 = 30; const P_ARTIST_MORE: i64 = 40; const P_SEARCH: i64 = 60; @@ -144,6 +147,24 @@ impl Crawler { } } + /// An artist radio was opened: crawl the artist, and the related pages of their + /// best tracks that were not read yet, to fill in the artist graph around them. + pub async fn explore_artist(&self, handle: String) { + self.push(P_FEEDBACK, Job::Artist(handle.clone())); + let h = handle.clone(); + let tracks = self + .db + .call(move |c| { + c.prepare("SELECT id, slug FROM tracks WHERE handle=? AND related_fetched=0 ORDER BY saves + downloads DESC LIMIT 3")? + .query_map([&h], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, String>(1)?)))? + .collect::, _>>() + }) + .await; + for (id, slug) in tracks.unwrap_or_default() { + self.push(P_FEEDBACK, Job::Related(id, handle.clone(), slug)); + } + } + // Main loop ------------------------------------------------------------- pub fn start(self: &Arc) { @@ -181,6 +202,9 @@ impl Crawler { self.push(P_SEED, Job::Home); self.push(P_SEED, Job::Ticker); self.push(P_SEED, Job::BestArtists); + for term in STYLE_TERMS { + self.push(P_STYLE, Job::Search(term.to_string())); + } for term in SEARCH_TERMS { self.push(P_SEARCH, Job::Search(term.to_string())); } @@ -206,8 +230,9 @@ impl Crawler { )?; let related: Vec<(i64, String, String)> = c .prepare( - "SELECT id, handle, slug FROM tracks WHERE related_fetched=0 \ - ORDER BY saves + downloads DESC LIMIT 10", + "SELECT id, handle, slug FROM tracks t WHERE related_fetched=0 ORDER BY \ + EXISTS (SELECT 1 FROM track_tags g WHERE g.track_id = t.id AND g.tag = 'riddim') DESC, \ + saves + downloads DESC LIMIT 10", )? .query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))? .collect::>()?; @@ -312,6 +337,8 @@ impl Crawler { let rows = normalize::related(&d); let artists: Vec<_> = rows.iter().map(|r| (r.handle.clone(), r.artist_name.clone(), r.artist_img.clone(), r.artist_verified)).collect(); self.upsert(rows, Some(track_id)).await?; + let handles: Vec = artists.iter().map(|a| a.0.clone()).collect(); + self.db.call(move |c| db::add_related(c, track_id, &handles)).await?; for (h, name, img, verified) in artists { self.add_artist(h, "related", Some(name), img, verified, P_RELATED).await?; } @@ -384,7 +411,8 @@ impl Crawler { let Some(a) = item.get("artist") else { continue }; if let Some(h) = s(a, "handle") { let verified = a.get("isVerified").and_then(Value::as_bool).unwrap_or(false); - let prio = if verified { P_ARTIST_FIRST } else { P_SEARCH + 5 }; + let riddim = s(a, "name").into_iter().chain([h.clone()]).any(|n| n.to_lowercase().contains("riddim")); + let prio = if verified || riddim { P_ARTIST_FIRST } else { P_SEARCH + 5 }; self.add_artist(h, "search", s(a, "name"), s(a, "img"), verified, prio).await?; } } diff --git a/backend/src/db.rs b/backend/src/db.rs index e08a65d..edef170 100644 --- a/backend/src/db.rs +++ b/backend/src/db.rs @@ -107,6 +107,29 @@ CREATE TABLE IF NOT EXISTS follows ( ); "#; +// track_tags: see tags.rs. track_related: the artists on a track's "related" page; +// the artist graph is read from it. meta: small settings, like the tagger version. +const SCHEMA_V4: &str = r#" +CREATE TABLE IF NOT EXISTS track_tags ( + track_id INTEGER NOT NULL, + tag TEXT NOT NULL, + PRIMARY KEY (track_id, tag) +) WITHOUT ROWID; +CREATE INDEX IF NOT EXISTS track_tags_tag ON track_tags(tag, track_id); +CREATE TABLE IF NOT EXISTS track_related ( + track_id INTEGER NOT NULL, + handle TEXT NOT NULL, + PRIMARY KEY (track_id, handle) +) WITHOUT ROWID; +CREATE INDEX IF NOT EXISTS track_related_handle ON track_related(handle); +INSERT OR IGNORE INTO track_related (track_id, handle) + SELECT related_of, handle FROM tracks WHERE related_of IS NOT NULL; +CREATE TABLE IF NOT EXISTS meta ( + key TEXT PRIMARY KEY, + value +); +"#; + pub fn now() -> f64 { SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs_f64() } @@ -116,11 +139,15 @@ pub struct Db(Arc>); impl Db { pub fn open(path: &Path) -> anyhow::Result { - let conn = Connection::open(path)?; + let mut conn = Connection::open(path)?; conn.pragma_update(None, "journal_mode", "WAL")?; conn.pragma_update(None, "foreign_keys", "ON")?; conn.busy_timeout(std::time::Duration::from_secs(5))?; migrate(&conn)?; + let tagged = retag_if_stale(&mut conn)?; + if tagged > 0 { + tracing::info!("tagged {tagged} tracks"); + } Ok(Db(Arc::new(Mutex::new(conn)))) } @@ -162,9 +189,104 @@ fn migrate(conn: &Connection) -> rusqlite::Result<()> { if version < 3 { conn.execute_batch(&format!("BEGIN; {SCHEMA_V3} PRAGMA user_version = 3; COMMIT;"))?; } + if version < 4 { + conn.execute_batch(&format!("BEGIN; {SCHEMA_V4} PRAGMA user_version = 4; COMMIT;"))?; + } Ok(()) } +/// An empty in-memory database, for tests. +#[cfg(test)] +pub fn memory() -> Connection { + let c = Connection::open_in_memory().unwrap(); + migrate(&c).unwrap(); + c +} + +// Tags and the artist graph --------------------------------------------------- + +fn tags_of(t: &Track) -> Vec<&'static str> { + crate::tags::of(&t.title, t.description.as_deref(), &t.artist_name, &t.handle, t.bpm) +} + +fn set_tags(conn: &Connection, track_id: i64, tags: &[&str]) -> rusqlite::Result<()> { + conn.prepare_cached("DELETE FROM track_tags WHERE track_id=?")?.execute([track_id])?; + let mut insert = conn.prepare_cached("INSERT INTO track_tags (track_id, tag) VALUES (?,?)")?; + for tag in tags { + insert.execute(params![track_id, tag])?; + } + Ok(()) +} + +/// Tag the whole catalogue again if the rules in tags.rs changed. Returns how many +/// tracks were tagged. +pub fn retag_if_stale(conn: &mut Connection) -> rusqlite::Result { + let done: Option = conn.query_row("SELECT value FROM meta WHERE key='tagger'", [], |r| r.get(0)).optional()?; + if done == Some(crate::tags::VERSION) { + return Ok(0); + } + let tx = conn.transaction()?; + let tracks: Vec = tx.prepare("SELECT * FROM tracks")?.query_map([], Track::from_row)?.collect::>()?; + tx.execute("DELETE FROM track_tags", [])?; + for t in &tracks { + set_tags(&tx, t.id, &tags_of(t))?; + } + tx.execute("INSERT OR REPLACE INTO meta (key, value) VALUES ('tagger', ?)", [crate::tags::VERSION])?; + tx.commit()?; + Ok(tracks.len()) +} + +pub fn track_tags(conn: &Connection, track_id: i64) -> rusqlite::Result> { + conn.prepare_cached("SELECT tag FROM track_tags WHERE track_id=? ORDER BY tag")?.query_map([track_id], |r| r.get(0))?.collect() +} + +/// Remember which artists showed up on a track's related page. +pub fn add_related(conn: &mut Connection, track_id: i64, handles: &[String]) -> rusqlite::Result<()> { + let tx = conn.transaction()?; + { + let mut insert = tx.prepare_cached("INSERT OR IGNORE INTO track_related (track_id, handle) VALUES (?,?)")?; + for h in handles { + insert.execute(params![track_id, h])?; + } + } + tx.commit() +} + +/// Artist graph edges: (artist, related artist, how many of the first one's tracks +/// listed the second). Self links are left out. +pub fn artist_links(conn: &Connection) -> rusqlite::Result> { + conn.prepare_cached( + "SELECT t.handle, r.handle, COUNT(*) FROM track_related r JOIN tracks t ON t.id = r.track_id \ + WHERE t.handle <> r.handle GROUP BY t.handle, r.handle", + )? + .query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))? + .collect() +} + +/// Artists linked to `handle` in either direction, with the summed link counts. +pub fn artist_neighbours(conn: &Connection, handle: &str) -> rusqlite::Result> { + conn.prepare_cached( + "SELECT h, SUM(n) FROM ( \ + SELECT r.handle AS h, COUNT(*) AS n FROM track_related r JOIN tracks t ON t.id = r.track_id \ + WHERE t.handle = ?1 AND r.handle <> ?1 GROUP BY r.handle \ + UNION ALL \ + SELECT t.handle, COUNT(*) FROM track_related r JOIN tracks t ON t.id = r.track_id \ + WHERE r.handle = ?1 AND t.handle <> ?1 GROUP BY t.handle \ + ) GROUP BY h", + )? + .query_map([handle], |r| Ok((r.get(0)?, r.get(1)?)))? + .collect() +} + +/// How many tracks each artist has with `tag`. +pub fn tag_counts(conn: &Connection, tag: &str) -> rusqlite::Result> { + conn.prepare_cached( + "SELECT t.handle, COUNT(*) FROM track_tags g JOIN tracks t ON t.id = g.track_id WHERE g.tag=? GROUP BY t.handle", + )? + .query_map([tag], |r| Ok((r.get(0)?, r.get(1)?)))? + .collect() +} + // Tracks ------------------------------------------------------------------ /// One row of `tracks`, as normalised from pl8list or read back. @@ -300,6 +422,7 @@ pub fn upsert_tracks(conn: &mut Connection, rows: &[Track], related_of: Option rusqlite::Result<(usize, usize)> { mod tests { use super::*; - fn mem() -> Connection { - let c = Connection::open_in_memory().unwrap(); - migrate(&c).unwrap(); - c + use super::memory as mem; + + #[test] + fn upsert_tags_tracks_and_retag_follows_the_tagger_version() { + let mut c = mem(); + let t = Track { id: 1, handle: "h".into(), artist_name: "H".into(), slug: "s".into(), title: "Big VIP".into(), bpm: Some(140.0), ..Default::default() }; + upsert_tracks(&mut c, &[t], None).unwrap(); + assert_eq!(track_tags(&c, 1).unwrap(), ["140", "vip"]); + c.execute("DELETE FROM track_tags", []).unwrap(); + assert_eq!(retag_if_stale(&mut c).unwrap(), 1); + assert_eq!(retag_if_stale(&mut c).unwrap(), 0); + assert_eq!(track_tags(&c, 1).unwrap(), ["140", "vip"]); + } + + #[test] + fn artist_graph_reads_related_pages_both_ways() { + let mut c = mem(); + let track = |id, h: &str| Track { id, handle: h.into(), artist_name: h.into(), slug: format!("s{id}"), title: "T".into(), ..Default::default() }; + upsert_tracks(&mut c, &[track(1, "a"), track(2, "a"), track(3, "b")], None).unwrap(); + add_related(&mut c, 1, &["b".into(), "c".into(), "a".into()]).unwrap(); + add_related(&mut c, 2, &["b".into()]).unwrap(); + add_related(&mut c, 3, &["d".into()]).unwrap(); + let mut n = artist_neighbours(&c, "a").unwrap(); + n.sort(); + assert_eq!(n, [("b".to_string(), 2), ("c".to_string(), 1)]); + let mut n = artist_neighbours(&c, "b").unwrap(); + n.sort(); + assert_eq!(n, [("a".to_string(), 2), ("d".to_string(), 1)]); + assert_eq!(artist_links(&c).unwrap().len(), 3); } #[test] diff --git a/backend/src/http/app.rs b/backend/src/http/app.rs index 95189c7..157c39e 100644 --- a/backend/src/http/app.rs +++ b/backend/src/http/app.rs @@ -61,9 +61,14 @@ pub async fn feed(State(app): St, Extension(current): Extension, Qu Ok(u) => u.clone(), Err(r) => return Ok(r), }; - let Some(ranker) = ranker::by_name(q.get("mode").map(String::as_str)) else { + let artist = q.get("artist").map(String::as_str).filter(|h| valid_handle(h)); + let Some(mode) = ranker::Mode::parse(q.get("mode").map(String::as_str), artist) else { return Ok(json_error(StatusCode::BAD_REQUEST, "unknown mode")); }; + // The artist graph is thin at first: crawl the radio's artist and their related pages. + if let ranker::Mode::Artist(h) = &mode { + app.crawler.explore_artist(h.clone()).await; + } let n = num(&q, "n").map(|n| n as usize).unwrap_or(10).clamp(1, 30); let filters = ranker::Filters { free_only: q.get("free").map(String::as_str) == Some("1"), @@ -74,7 +79,8 @@ pub async fn feed(State(app): St, Extension(current): Extension, Qu let deadline = Instant::now() + FEED_WAIT; let uid = user.id; let mut tracks = loop { - let tracks = app.db.call(move |c| ranker::pick(c, uid, n, filters, ranker)).await?; + let mode = mode.clone(); + let tracks = app.db.call(move |c| ranker::pick(c, uid, n, filters, &*mode.ranker(c)?)).await?; if !tracks.is_empty() || Instant::now() >= deadline { break tracks; } diff --git a/backend/src/http/tests.rs b/backend/src/http/tests.rs index aeb556c..e6c7a1c 100644 --- a/backend/src/http/tests.rs +++ b/backend/src/http/tests.rs @@ -380,3 +380,40 @@ async fn token_rotation_keeps_the_pl8list_connection() { assert_eq!(app.sync.key.sealer(&new, 5).open(&new_blob.unwrap()).unwrap().expose_secret(), "PL8"); assert!(old_blob.is_none()); } + +#[tokio::test] +async fn feed_modes_filter_and_tag_tracks() { + let (r, app, _) = setup("modes", "http://127.0.0.1:9").await; + let token = app + .db + .call(|c| { + let t = |id, title: &str, bpm| crate::db::Track { + id, + handle: "h".into(), + artist_name: "H".into(), + slug: format!("s{id}"), + title: title.into(), + bpm: Some(bpm), + full_access: true, + ..Default::default() + }; + crate::db::upsert_tracks(c, &[t(1, "Riddim VIP", 140.0), t(2, "Roller", 174.0)], None)?; + crate::db::upsert_user(c, 5, "tester", None, None)?; + crate::db::create_session(c, 5) + }) + .await + .unwrap(); + let cookie = format!("__Host-pl8inf={token}"); + let get = |uri: &str| r.clone().oneshot(req("GET", uri, None, Some(&cookie), "203.0.113.1")); + let res = get("/local/feed?mode=riddim").await.unwrap(); + assert_eq!(res.status(), StatusCode::OK); + let body = json_body(res).await; + let tracks = body["tracks"].as_array().unwrap(); + assert_eq!(tracks.len(), 1); + assert_eq!(tracks[0]["tags"], serde_json::json!(["140", "riddim", "vip"])); + for bad in ["/local/feed?mode=nope", "/local/feed?mode=artist", "/local/feed?mode=artist&artist=..%2Fx"] { + assert_eq!(get(bad).await.unwrap().status(), StatusCode::BAD_REQUEST, "{bad}"); + } + let res = get("/local/feed?mode=artist&artist=h").await.unwrap(); + assert_eq!(json_body(res).await["tracks"].as_array().unwrap().len(), 1); +} diff --git a/backend/src/main.rs b/backend/src/main.rs index 34e7788..1ce7058 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -23,6 +23,7 @@ mod pl8; mod ranker; mod ratelimit; mod sync; +mod tags; use std::{ io::{Read, Write}, diff --git a/backend/src/ranker.rs b/backend/src/ranker.rs index dc02947..4c5d696 100644 --- a/backend/src/ranker.rs +++ b/backend/src/ranker.rs @@ -11,8 +11,8 @@ //! exploration Gumbel noise, so the feed never freezes into one niche //! //! The batch is then picked greedily with a penalty for repeating an artist. -//! A `Ranker` supplies the weights and may narrow the candidates; discovery modes -//! are further `Ranker`s chosen with `?mode=`. +//! A `Ranker` supplies the weights, may narrow the candidates and may add a bonus. +//! Discovery modes (`Mode`) are further `Ranker`s chosen with `?mode=`. use std::collections::{HashMap, HashSet}; @@ -40,6 +40,14 @@ pub trait Ranker: Send + Sync { fn condition(&self) -> Option<&str> { None } + /// Drop candidates the SQL condition cannot express. + fn keep(&self, _track: &Track) -> bool { + true + } + /// Added to a candidate's score. + fn bonus(&self, _track: &Track) -> f64 { + 0.0 + } } /// The original mix. @@ -62,11 +70,161 @@ impl Ranker for Classic { } } -pub fn by_name(mode: Option<&str>) -> Option<&'static dyn Ranker> { - match mode.unwrap_or("") { - "" | "classic" => Some(&Classic), - _ => None, +/// Discovery modes, chosen with `?mode=` (and `&artist=` for an artist radio). +#[derive(Clone, Debug, PartialEq)] +pub enum Mode { + Classic, + /// 140 BPM tracks, riddim first: tagged riddim, or by artists who make it or sit + /// next to them in the artist graph. + Riddim, + /// Little-known tracks: popularity counts against a track. + Deep, + /// Uploads of the last few weeks. + Fresh, + /// "More like this artist": the artist and their neighbours in the artist graph. + Artist(String), +} + +impl Mode { + pub fn parse(mode: Option<&str>, artist: Option<&str>) -> Option { + Some(match mode.unwrap_or("") { + "" | "classic" => Mode::Classic, + "riddim" => Mode::Riddim, + "deep" => Mode::Deep, + "fresh" => Mode::Fresh, + "artist" => Mode::Artist(artist?.to_string()), + _ => return None, + }) } + + pub fn ranker(&self, conn: &Connection) -> rusqlite::Result> { + Ok(match self { + Mode::Classic => Box::new(Classic), + Mode::Riddim => Box::new(Riddim { affinity: tag_affinity(conn, "riddim")? }), + Mode::Deep => Box::new(Deep), + Mode::Fresh => Box::new(Fresh), + Mode::Artist(h) => Box::new(ArtistRadio { seed: h.clone(), near: neighbourhood(conn, h)? }), + }) + } +} + +struct Riddim { + affinity: HashMap, +} + +const RIDDIM: Weights = Weights { popularity: 0.8, ..CLASSIC }; + +impl Ranker for Riddim { + fn weights(&self) -> &Weights { + &RIDDIM + } + fn condition(&self) -> Option<&str> { + Some("EXISTS (SELECT 1 FROM track_tags g WHERE g.track_id = t.id AND g.tag = '140')") + } + fn bonus(&self, t: &Track) -> f64 { + 2.0 * self.affinity.get(&t.handle).copied().unwrap_or(0.0) + } +} + +struct Deep; + +const DEEP: Weights = Weights { + popularity: -0.8, + freshness: 0.3, + artist: 0.6, + tempo: 1.0, + key: 0.3, + related: 0.5, + unknown_artist: 1.0, + exploration: 0.8, +}; + +impl Ranker for Deep { + fn weights(&self) -> &Weights { + &DEEP + } + fn condition(&self) -> Option<&str> { + Some("t.saves <= 15") + } +} + +struct Fresh; + +const FRESH: Weights = Weights { freshness: 3.0, popularity: 0.6, ..CLASSIC }; + +impl Ranker for Fresh { + fn weights(&self) -> &Weights { + &FRESH + } + // pl8list dates look like 2026-10-31T18:34:13.657Z, so they compare as text. + fn condition(&self) -> Option<&str> { + Some("t.created_at >= strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-45 days')") + } +} + +struct ArtistRadio { + seed: String, + /// Neighbours, 0..1 by how closely they are linked. + near: HashMap, +} + +const RADIO: Weights = Weights { artist: 0.5, related: 0.5, ..CLASSIC }; + +impl Ranker for ArtistRadio { + fn weights(&self) -> &Weights { + &RADIO + } + fn keep(&self, t: &Track) -> bool { + t.handle == self.seed || self.near.contains_key(&t.handle) + } + // The artist themselves now and then; mostly the ones around them. + fn bonus(&self, t: &Track) -> f64 { + if t.handle == self.seed { 1.0 } else { 2.0 * self.near.get(&t.handle).copied().unwrap_or(0.0) } + } +} + +/// Neighbours of an artist up to two hops away, scored 0..1. A second-hop artist +/// scores at most 0.4 of the link that leads to it. +fn neighbourhood(conn: &Connection, seed: &str) -> rusqlite::Result> { + let scaled = |links: Vec<(String, i64)>| { + let max = links.iter().map(|l| l.1).max().unwrap_or(1).max(1) as f64; + links.into_iter().map(|(h, n)| (h, (n as f64 / max).sqrt())).collect::>() + }; + let first = scaled(crate::db::artist_neighbours(conn, seed)?); + let mut near: HashMap = first.iter().cloned().collect(); + let mut by_score = first; + by_score.sort_by(|a, b| b.1.total_cmp(&a.1)); + for (h, w) in by_score.into_iter().take(10) { + for (h2, w2) in scaled(crate::db::artist_neighbours(conn, &h)?) { + if h2 != seed { + let s = near.entry(h2).or_default(); + *s = s.max(0.4 * w * w2); + } + } + } + Ok(near) +} + +/// How strongly each artist is linked to `tag`, 0..1: their own tagged tracks (three +/// or more count fully), or half of what the artists next to them in the graph have. +pub fn tag_affinity(conn: &Connection, tag: &str) -> rusqlite::Result> { + let own: HashMap = crate::db::tag_counts(conn, tag)?.into_iter().map(|(h, n)| (h, (n as f64 / 3.0).min(1.0))).collect(); + let mut spread: HashMap = HashMap::new(); + for (a, b, n) in crate::db::artist_links(conn)? { + let n = n as f64; + for (from, to) in [(&a, &b), (&b, &a)] { + let e = spread.entry(to.clone()).or_default(); + e.0 += n * own.get(from).copied().unwrap_or(0.0); + e.1 += n; + } + } + let mut out = own; + for (h, (sum, total)) in spread { + let s = out.entry(h).or_default(); + *s = s.max(0.5 * sum / total); + } + out.retain(|_, v| *v > 0.0); + Ok(out) } /// Per-event contributions to artist affinity. @@ -183,7 +341,8 @@ pub fn pick(conn: &Connection, user_id: i64, n: usize, f: Filters, ranker: &dyn sql.push_str(c); sql.push(')'); } - let cands: Vec = conn.prepare(&sql)?.query_map(params_from_iter(args), Track::from_row)?.collect::>()?; + let mut cands: Vec = conn.prepare(&sql)?.query_map(params_from_iter(args), Track::from_row)?.collect::>()?; + cands.retain(|t| ranker.keep(t)); if cands.is_empty() { return Ok(vec![]); } @@ -217,7 +376,8 @@ pub fn pick(conn: &Connection, user_id: i64, n: usize, f: Filters, ranker: &dyn + w.key * if r.key.as_ref().is_some_and(|k| t.keys.contains(k)) { 1.0 } else { 0.0 } + w.related * if r.related_of.is_some_and(|id| t.positive_ids.contains(&id)) { 1.0 } else { 0.0 } + w.unknown_artist * if aff.is_none() { 1.0 } else { 0.0 } - + w.exploration * -(-u.ln()).ln(); + + w.exploration * -(-u.ln()).ln() + + ranker.bonus(&r); (s, r) }) .collect(); @@ -233,9 +393,19 @@ pub fn pick(conn: &Connection, user_id: i64, n: usize, f: Filters, ranker: &dyn seen.push(r.handle.clone()); out.push(serialize(&r, Some(s))); } + add_tags(conn, &mut out)?; Ok(out) } +fn add_tags(conn: &Connection, tracks: &mut [Value]) -> rusqlite::Result<()> { + for t in tracks { + if let Some(id) = t["id"].as_i64() { + t["tags"] = json!(crate::db::track_tags(conn, id)?); + } + } + Ok(()) +} + pub fn serialize(r: &Track, score: Option) -> Value { let preview = !r.full_access; let wave = if preview && r.preview_waveform.is_some() { &r.preview_waveform } else { &r.waveform }; @@ -278,6 +448,7 @@ pub fn likes(conn: &Connection, user_id: i64) -> rusqlite::Result> { out.push(serialize(&t, None)); } } + add_tags(conn, &mut out)?; Ok(out) } @@ -285,6 +456,84 @@ pub fn likes(conn: &Connection, user_id: i64) -> rusqlite::Result> { mod tests { use super::*; + use crate::db; + + fn track(id: i64, handle: &str, title: &str, bpm: f64, saves: i64) -> Track { + Track { + id, + handle: handle.into(), + artist_name: handle.into(), + slug: format!("s{id}"), + title: title.into(), + bpm: Some(bpm), + saves, + full_access: true, + created_at: Some("2020-01-01T00:00:00.000Z".into()), + ..Default::default() + } + } + + fn ids(conn: &Connection, mode: Mode, n: usize) -> Vec { + let r = mode.ranker(conn).unwrap(); + let mut out: Vec = pick(conn, 1, n, Filters::default(), &*r).unwrap().iter().map(|t| t["id"].as_i64().unwrap()).collect(); + out.sort(); + out + } + + #[test] + fn riddim_mode_keeps_140_and_ranks_riddim_artists_and_their_neighbours_first() { + let mut c = db::memory(); + let rows = [ + track(1, "rid", "Wobble riddim", 140.0, 0), + track(2, "rid", "Other", 140.0, 0), + track(3, "near", "Plain", 140.0, 0), + track(4, "far", "Plain", 140.0, 500), + track(5, "dnb", "Roller", 174.0, 900), + ]; + db::upsert_tracks(&mut c, &rows, None).unwrap(); + db::add_related(&mut c, 3, &["rid".into()]).unwrap(); + let aff = tag_affinity(&c, "riddim").unwrap(); + assert!((aff["rid"] - 1.0 / 3.0).abs() < 1e-9); + assert!(aff["near"] > 0.0 && aff["near"] < aff["rid"]); + assert!(!aff.contains_key("far")); + assert_eq!(ids(&c, Mode::Riddim, 10), [1, 2, 3, 4]); + let r = Mode::Riddim.ranker(&c).unwrap(); + assert!(r.bonus(&rows[0]) > r.bonus(&rows[2]) && r.bonus(&rows[2]) > r.bonus(&rows[3])); + } + + #[test] + fn artist_radio_stays_near_the_artist() { + let mut c = db::memory(); + let rows = [track(1, "a", "T", 140.0, 0), track(2, "b", "T", 140.0, 0), track(3, "c", "T", 140.0, 0), track(4, "z", "T", 140.0, 0)]; + db::upsert_tracks(&mut c, &rows, None).unwrap(); + db::add_related(&mut c, 1, &["b".into()]).unwrap(); + db::add_related(&mut c, 2, &["c".into()]).unwrap(); + let near = neighbourhood(&c, "a").unwrap(); + assert_eq!(near["b"], 1.0); + assert!((near["c"] - 0.4).abs() < 1e-9); + assert_eq!(ids(&c, Mode::Artist("a".into()), 10), [1, 2, 3]); + } + + #[test] + fn deep_and_fresh_narrow_the_catalogue() { + let mut c = db::memory(); + let mut new = track(3, "c", "T", 140.0, 0); + new.created_at = Some(c.query_row("SELECT strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day')", [], |r| r.get(0)).unwrap()); + db::upsert_tracks(&mut c, &[track(1, "a", "T", 140.0, 2), track(2, "b", "T", 140.0, 400), new], None).unwrap(); + assert_eq!(ids(&c, Mode::Deep, 10), [1, 3]); + assert_eq!(ids(&c, Mode::Fresh, 10), [3]); + assert_eq!(ids(&c, Mode::Classic, 10), [1, 2, 3]); + } + + #[test] + fn modes_parse() { + assert_eq!(Mode::parse(None, None), Some(Mode::Classic)); + assert_eq!(Mode::parse(Some("riddim"), None), Some(Mode::Riddim)); + assert_eq!(Mode::parse(Some("artist"), Some("x")), Some(Mode::Artist("x".into()))); + assert_eq!(Mode::parse(Some("artist"), None), None); + assert_eq!(Mode::parse(Some("nope"), None), None); + } + #[test] fn tempo_counts_half_and_double_time() { assert!(tempo_match(Some(140.0), &[140.0]) > 0.99); diff --git a/backend/src/tags.rs b/backend/src/tags.rs new file mode 100644 index 0000000..0758a05 --- /dev/null +++ b/backend/src/tags.rs @@ -0,0 +1,105 @@ +//! Tags for a track, read from its title, description, artist name and BPM. +//! +//! pl8list has no genre field, so this is keyword work. Producers name their files +//! after what they are ("… VIP", "… [DUBPACK]", "… (X FLIP) v2.wav"), which says a +//! lot about versions, and a little about style. Style tags are sparse: most riddim +//! is not called riddim. `ranker::tag_affinity` spreads them from tagged tracks to +//! their artists and the artists pl8list relates them to. +//! +//! Bump `VERSION` when the rules change; the catalogue is retagged on the next start. + +pub const VERSION: i64 = 2; + +fn words(text: &str) -> Vec { + text.split(|c: char| !c.is_alphanumeric()) + .filter(|w| !w.is_empty()) + .map(str::to_lowercase) + .collect() +} + +/// Tags of one track, sorted and without duplicates. +pub fn of(title: &str, description: Option<&str>, artist: &str, handle: &str, bpm: Option) -> Vec<&'static str> { + let amp = title.contains(" & "); + // File names: "x_y-z.wav" splits into words like any other title. + let title = words(title); + let mut text = title.clone(); + text.extend(words(description.unwrap_or(""))); + let has = |w: &str| text.iter().any(|t| t == w); + let pair = |a: &[&str], b: &[&str]| text.windows(2).any(|p| a.contains(&p[0].as_str()) && b.contains(&p[1].as_str())); + let triple = |a: &str, b: &[&str], c: &str| text.windows(3).any(|p| p[0] == a && b.contains(&p[1].as_str()) && p[2] == c); + + let dnb = has("dnb") || pair(&["drum"], &["bass"]) || triple("drum", &["and", "n"], "bass"); + + let mut tags = vec![]; + let mut tag = |on: bool, t: &'static str| { + if on { + tags.push(t); + } + }; + + let name = format!("{artist} {handle}").to_lowercase(); + tag(has("riddim") || has("riddims") || has("riddimz") || name.contains("riddim"), "riddim"); + tag(has("tearout") || pair(&["tear"], &["out"]), "tearout"); + tag(has("deathstep"), "deathstep"); + tag(has("brostep"), "brostep"); + tag(has("colourbass") || has("colorbass") || pair(&["colour", "color"], &["bass"]), "colourbass"); + tag(has("melodic"), "melodic"); + tag(has("wonk") || has("wonky"), "wonk"); + tag(has("halftime") || pair(&["half"], &["time"]), "halftime"); + tag(dnb, "dnb"); + tag(has("jungle"), "jungle"); + tag(has("garage") || has("ukg") || has("2step"), "garage"); + tag(has("grime"), "grime"); + tag(has("trap"), "trap"); + + tag(has("vip"), "vip"); + tag(has("dubpack") || pair(&["dub"], &["pack"]), "dubpack"); + tag(has("dub") && !pair(&["dub"], &["pack"]), "dub"); + tag(has("remix") || has("rmx"), "remix"); + tag(has("flip"), "flip"); + tag(has("bootleg"), "bootleg"); + tag(has("edit"), "edit"); + tag(has("spesh") || has("special"), "special"); + let between = |w: &str| title.len() > 2 && title[1..title.len() - 1].iter().any(|t| t == w); + tag(has("collab") || has("ft") || has("feat") || between("x") || between("vs") || amp && !dnb, "collab"); + tag(has("wip") || has("unfinished") || text.iter().any(|w| w.len() > 1 && w.starts_with('v') && w[1..].bytes().all(|b| b.is_ascii_digit())), "wip"); + + if let Some(b) = bpm.filter(|b| *b > 0.0) { + tag((138.0..=152.0).contains(&b) || (69.0..=76.0).contains(&b), "140"); + tag((165.0..=180.0).contains(&b) || (82.5..=90.0).contains(&b), "170"); + } + tags.sort_unstable(); + tags.dedup(); + tags +} + +#[cfg(test)] +mod tests { + use super::*; + + fn t(title: &str) -> Vec<&'static str> { + of(title, None, "Someone", "someone", None) + } + + #[test] + fn reads_versions_from_file_names() { + assert_eq!(t("THRAX x JADHA - MR.WAYNE V1.wav"), ["collab", "wip"]); + assert_eq!(t("4. VIRTU - GANG VIP [THE VAULT DUBPACK VOL. 1].wav"), ["dubpack", "vip"]); + assert_eq!(t("KOH - FRACTURE [Cmadden SPESH].wav"), ["special"]); + assert_eq!(t("MOTUS & LOWER - FEENZ"), ["collab"]); + assert_eq!(t("LV VS 5CHIZO - HARDIONICFLOW.wav"), ["collab"]); + assert_eq!(t("Drum & Bass Roller"), ["dnb"]); + assert_eq!(t("Riddimo - Stacked Vip v1.mp3"), ["vip", "wip"]); + assert_eq!(t("SOMETHING (DUB)"), ["dub"]); + // Words inside other words do not count. + assert_eq!(t("VIPER - EDITION X"), Vec::<&str>::new()); + } + + #[test] + fn style_from_text_and_artist() { + assert_eq!(of("Track", Some("#Khmer #DNB #174bpm"), "A", "a", Some(174.0)), ["170", "dnb"]); + assert_eq!(of("Heavy one", None, "Riddim Lord", "lord", Some(70.0)), ["140", "riddim"]); + assert_eq!(of("tear out weapon", Some("drum n bass"), "A", "a", None), ["dnb", "tearout"]); + assert_eq!(of("x", None, "A", "a", Some(100.0)), Vec::<&str>::new()); + } +} diff --git a/backend/templates/datenschutz.html b/backend/templates/datenschutz.html index b12062f..5291cc8 100644 --- a/backend/templates/datenschutz.html +++ b/backend/templates/datenschutz.html @@ -134,7 +134,7 @@
  • __Host-pl8inf: Ihre Sitzung, bis zu 30 Tage ohne Nutzung. Wenn Sie verbunden bleiben, ist es auch Teil des Schlüssels für Ihre pl8list-Sitzung (Abschnitt 5).
  • __Host-pl8inf_login: verbindet die beiden Schritte der Anmeldung, 15 Minuten.
    • -
  • Im lokalen Speicher Ihres Browsers: Ihre Filter (z. B. BPM-Bereich) und ob Sie den Bedienhinweis schon gesehen haben.
    • +
  • Im lokalen Speicher Ihres Browsers: Ihre Filter (z. B. BPM-Bereich), der gewählte Feed (z. B. „riddim“ oder ein Künstler-Radio) und ob Sie den Bedienhinweis schon gesehen haben.

Alle sind für den von Ihnen gewünschten Dienst unbedingt erforderlich (§ 25 Abs. 2 Nr. 2 TDDDG). diff --git a/web/index.html b/web/index.html index 12217f7..3381dc7 100644 --- a/web/index.html +++ b/web/index.html @@ -31,6 +31,7 @@