integrity: one write, one transaction, and nothing left behind

Node update, delete and move each ran as separate statements, with the
slug chosen outside the write that took it. They are now one transaction
each, opened with BEGIN IMMEDIATE so a read-then-write cannot fail with
SQLITE_BUSY halfway. Creating a node, a space or a file does the same.

A delete is one function, delete_subtree, used by the API and the SFTP
mount alike: SFTP deletes used to skip the media cleanup and orphan the
files a document embedded. Rows go in the transaction, bytes after the
commit. Deleting a media row that backs a file in the tree is refused;
the media browser marks those files and stops counting them as
documents that embed them.

A space can no longer be moved into a folder. The diary is found by a
role column (migration 0013) instead of its slug, so renaming it no
longer breaks the reminder or the agenda, and it cannot be deleted. The
same migration adds the unique index root slugs never had.

A card is done exactly when it is in the last list: a done patch now
moves the card, a card made in the done list is stamped, and adding or
deleting lists settles the board. Seeding a fresh board is atomic, and
a card links only to a document.

Uploads are fsynced before the rename, and zip entries are written aside
and renamed like any other upload. A sweep at start and once a day
removes leftover .part files, files no row knows, and uploads nothing
has embedded for a week.

Before a new build migrates the database it writes a VACUUM INTO copy,
diary.db.pre-<version>, since migrations only go forward.

README.md

@@ -107,8 +107,9 @@ same). A new board starts with `backlog`, `doing` and `done`, and the columns me
 something by where they stand rather than by their names: the first is the
 inbox new cards land in, the last is done. A card is finished exactly when it
 sits in the last column — moving it there stamps it, moving it out reopens it —
-so a tick and a column can never disagree. `:list <name>` adds a column in front
+so a tick and a column can never disagree; marking a card done through the API
-of the last one, `:name!` on the board renames the column under the cursor, and
+moves it there too, and deleting the last column makes the one before it done.
+`:list <name>` adds a column in front of the last one, `:name!` on the board renames the column under the cursor, and
 done cards older than a week fold away until `z` shows them.
 
 `c` captures from anywhere: `:card <title>` outside the board puts the card in
@@ -135,7 +136,8 @@ and `Enter` on that card opens it again. `Enter` on a card without one makes a
 document named after the card in the space's root and opens it to write in, and
 closing that document returns to the board on the same card. The link is deliberately loose —
 deleting the document leaves the card standing, because the work is still open
-once its notes are gone.
+once its notes are gone. Only a document can be a card's notes; a folder or a
+file is refused.
 
 ## The agenda
 
@@ -161,6 +163,17 @@ document, every file in the tree where it sits and every other embedded file in
 `media/`, under their real names, and relative links between them, so the
 archive reads in any markdown viewer without this application.
 
+The diary is the diary by role, not by name: rename it and the reminder and
+the agenda still know which space it is, and it cannot be deleted — empty it
+instead. Before a new build migrates the database it keeps a copy of it as
+it was, `diary.db.pre-<version>`, next to it in the data directory.
+
+An upload that nothing embeds any more — pasted, then deleted from the
+document — is kept a week and then removed, along with what a killed upload
+or a crash leaves in `uploads/`. That runs at start and once a day. A file in
+the tree is never one of them: it is deleted from its folder, not from
+`:media`.
+
 ## Files
 
 A file is a node like a document, only with bytes instead of text: it sits in

migrations/0013_roles.sql

@@ -0,0 +1,12 @@
+-- What a space is for, where the code has to know. The diary used to be found
+-- by its slug, so renaming it — which moves the slug — left the reminder
+-- believing nothing had been written in it, ever. A role survives a rename.
+ALTER TABLE nodes ADD COLUMN role TEXT;
+UPDATE nodes SET role = 'diary'
+WHERE parent_id IS NULL AND kind = 'space' AND slug = 'diary';
+CREATE UNIQUE INDEX idx_nodes_role ON nodes (role) WHERE role IS NOT NULL;
+
+-- (parent_id, slug) is unique, but SQLite counts every NULL as distinct, so
+-- that index never covered the spaces themselves: root slugs were unique only
+-- because the code checked before it inserted.
+CREATE UNIQUE INDEX idx_nodes_root_slug ON nodes (slug) WHERE parent_id IS NULL;

src/db.rs

@@ -0,0 +1,14 @@
+//! What every writer to the database shares.
+
+use sqlx::{Sqlite, SqlitePool, Transaction};
+
+/// A transaction that takes the write lock up front. SQLite's default
+/// `BEGIN` is deferred: a transaction that reads before it writes holds only
+/// a read snapshot, and if another connection commits in between, the first
+/// write fails at once with `SQLITE_BUSY` — `busy_timeout` cannot help,
+/// because waiting would not make the snapshot current. Taking the lock at
+/// `BEGIN` makes the reads and the writes one consistent step, and turns
+/// contention into a wait instead of an error.
+pub async fn begin_write(db: &SqlitePool) -> sqlx::Result<Transaction<'static, Sqlite>> {
+    db.begin_with("BEGIN IMMEDIATE").await
+}

src/main.rs

@@ -2,6 +2,7 @@ mod auth;
 mod backup;
 mod cli;
 mod config;
+mod db;
 mod error;
 mod mail;
 mod notify;
@@ -9,6 +10,7 @@ mod routes;
 mod sftp;
 mod state;
 mod static_files;
+mod sweep;
 mod throttle;
 mod vault;
 
@@ -110,6 +112,8 @@ async fn main() -> Result<()> {
     let notify = notify::Notifier::new(db.clone(), Arc::clone(&config))?;
     notify::spawn(Arc::clone(&notify));
 
+    sweep::spawn(db.clone(), config.uploads_dir());
+
     let state = AppState {
         vault: vault::Vault::new(db.clone(), config.vault_idle),
         db,
@@ -157,7 +161,9 @@ async fn open_db(config: &Config) -> Result<sqlx::SqlitePool> {
         .await
         .context("could not open the workspace database")?;
 
-    sqlx::migrate!("./migrations")
+    let migrator = sqlx::migrate!("./migrations");
+    snapshot_before_migrating(&db, &migrator, config).await?;
+    migrator
         .run(&db)
         .await
         .context("could not apply database migrations")?;
@@ -165,6 +171,50 @@ async fn open_db(config: &Config) -> Result<sqlx::SqlitePool> {
     Ok(db)
 }
 
+/// A copy of the database as it was, taken before a migration changes it.
+/// Migrations only go forward, and some of them reshape or drop what was
+/// there; the copy is the way back if one goes wrong on real data. A fresh
+/// database, or one already up to date, gets none.
+async fn snapshot_before_migrating(
+    db: &sqlx::SqlitePool,
+    migrator: &sqlx::migrate::Migrator,
+    config: &Config,
+) -> Result<()> {
+    let applied: Option<i64> = match sqlx::query_scalar(
+        "SELECT MAX(version) FROM _sqlx_migrations WHERE success",
+    )
+    .fetch_one(db)
+    .await
+    {
+        Ok(version) => version,
+        // No bookkeeping table: nothing has ever been migrated, so there is
+        // nothing to keep.
+        Err(_) => return Ok(()),
+    };
+    let Some(applied) = applied else {
+        return Ok(());
+    };
+    let latest = migrator.iter().map(|m| m.version).max().unwrap_or(0);
+    if latest <= applied {
+        return Ok(());
+    }
+
+    let path = config.data_dir.join(format!("diary.db.pre-{latest:04}"));
+    if path.exists() {
+        return Ok(());
+    }
+    tracing::info!(
+        "database is at migration {applied}, the binary at {latest}: keeping a copy at {}",
+        path.display()
+    );
+    sqlx::query("VACUUM INTO ?1")
+        .bind(path.to_string_lossy().into_owned())
+        .execute(db)
+        .await
+        .with_context(|| format!("could not snapshot the database to {}", path.display()))?;
+    Ok(())
+}
+
 async fn serve(bind: std::net::SocketAddr, max_upload: usize, state: AppState) -> Result<()> {
     let app = Router::new()
         .nest("/api", routes::api_router(max_upload))

src/notify/mod.rs

@@ -389,7 +389,7 @@ pub(crate) async fn diary_written(db: &SqlitePool, tz: Tz, day: NaiveDate) -> sq
     let (start, end) = day_bounds(tz, day);
     let written: i64 = sqlx::query_scalar(
         "WITH RECURSIVE tree (id) AS (
-             SELECT id FROM nodes WHERE parent_id IS NULL AND slug = 'diary'
+             SELECT id FROM nodes WHERE role = 'diary'
              UNION ALL
              SELECT n.id FROM nodes n JOIN tree t ON n.parent_id = t.id
          )

src/routes/boards.rs

@@ -21,12 +21,12 @@ use axum::{
     Json,
 };
 use serde::{Deserialize, Serialize};
-use sqlx::{Row, SqlitePool};
+use sqlx::{Row, SqliteConnection, SqlitePool};
 
 use crate::{
     auth::Session,
     error::{AppError, AppResult},
-    routes::nodes::SPACE,
+    routes::nodes::{DOCUMENT, SPACE},
     state::AppState,
 };
 
@@ -155,7 +155,7 @@ pub async fn agenda(_: Session, State(state): State<AppState>) -> AppResult<Json
     let cards = agenda_cards(&state.db, crate::now() + (AGENDA_DAYS + 1) * 86400).await?;
 
     let has_diary: Option<i64> =
-        sqlx::query_scalar("SELECT id FROM nodes WHERE parent_id IS NULL AND slug = 'diary'")
+        sqlx::query_scalar("SELECT id FROM nodes WHERE role = 'diary'")
             .fetch_optional(&state.db)
             .await?;
     let diary_today = match has_diary {
@@ -290,9 +290,12 @@ async fn list_space(db: &SqlitePool, list_id: i64) -> AppResult<i64> {
 /// existed — or one whose board was switched on later — gets its columns the
 /// first time the board is looked at, which is the only moment it matters.
 async fn ensure_lists(db: &SqlitePool, space_id: i64) -> AppResult<()> {
+    // Count and seed under one write lock: two tabs opening a fresh board at
+    // once would otherwise both find it empty and seed it twice.
+    let mut tx = crate::db::begin_write(db).await?;
     let existing: i64 = sqlx::query("SELECT COUNT(*) AS n FROM board_lists WHERE space_id = ?1")
         .bind(space_id)
-        .fetch_one(db)
+        .fetch_one(&mut *tx)
         .await?
         .get("n");
     if existing > 0 {
@@ -308,12 +311,65 @@ async fn ensure_lists(db: &SqlitePool, space_id: i64) -> AppResult<()> {
         .bind(name)
         .bind(position as i64)
         .bind(now)
-        .execute(db)
+        .execute(&mut *tx)
         .await?;
     }
+    tx.commit().await?;
     Ok(())
 }
 
+/// Make every card on a board agree with where it stands: finished in the
+/// last list, open anywhere else. Moving a card keeps this by itself; this is
+/// for the changes that move the lists instead — one added after a lone list,
+/// the last one deleted. A board of one list has no "done" to agree with and
+/// is left as it is.
+async fn settle(db: &mut SqliteConnection, space_id: i64) -> AppResult<()> {
+    let lists: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM board_lists WHERE space_id = ?1")
+        .bind(space_id)
+        .fetch_one(&mut *db)
+        .await?;
+    let Some((last, _)) = last_list(&mut *db, space_id).await? else {
+        return Ok(());
+    };
+    if lists < 2 {
+        return Ok(());
+    }
+    sqlx::query(
+        "UPDATE cards SET done_at = NULL
+         WHERE done_at IS NOT NULL AND list_id <> ?2
+           AND list_id IN (SELECT id FROM board_lists WHERE space_id = ?1)",
+    )
+    .bind(space_id)
+    .bind(last)
+    .execute(&mut *db)
+    .await?;
+    sqlx::query("UPDATE cards SET done_at = ?2 WHERE done_at IS NULL AND list_id = ?1")
+        .bind(last)
+        .bind(crate::now())
+        .execute(&mut *db)
+        .await?;
+    Ok(())
+}
+
+/// A card links to a document — its notes — and to nothing else: a folder
+/// cannot be opened to write in, and a file has no text to hold them.
+async fn linkable(db: &SqlitePool, node_id: Option<i64>) -> AppResult<()> {
+    let Some(node_id) = node_id else {
+        return Ok(());
+    };
+    let kind: Option<String> = sqlx::query_scalar("SELECT kind FROM nodes WHERE id = ?1")
+        .bind(node_id)
+        .fetch_optional(db)
+        .await?;
+    match kind.as_deref() {
+        Some(DOCUMENT) => Ok(()),
+        Some(kind) => Err(AppError::BadRequest(format!(
+            "a card links to a document, not a {kind}"
+        ))),
+        None => Err(AppError::NotFound),
+    }
+}
+
 pub async fn board(
     _: Session,
     State(state): State<AppState>,
@@ -386,13 +442,13 @@ pub async fn create_list(
     // A new column goes in front of the last one, because the last one is
     // "done" and has to stay last for that to keep meaning anything. A board
     // down to a single list just grows a second one after it.
+    let mut tx = crate::db::begin_write(&state.db).await?;
     let count: i64 = sqlx::query("SELECT COUNT(*) AS n FROM board_lists WHERE space_id = ?1")
         .bind(space_id)
-        .fetch_one(&state.db)
+        .fetch_one(&mut *tx)
         .await?
         .get("n");
 
-    let mut tx = state.db.begin().await?;
     let position = match last_list(&mut *tx, space_id).await? {
         Some((last, position)) if count >= 2 => {
             sqlx::query("UPDATE board_lists SET position = position + 1 WHERE id = ?1")
@@ -414,6 +470,7 @@ pub async fn create_list(
     .bind(crate::now())
     .execute(&mut *tx)
     .await?;
+    settle(&mut tx, space_id).await?;
     tx.commit().await?;
 
     state.backup.signal();
@@ -452,9 +509,10 @@ pub async fn remove_list(
 ) -> AppResult<Json<Board>> {
     let space_id = list_space(&state.db, list_id).await?;
 
+    let mut tx = crate::db::begin_write(&state.db).await?;
     let remaining: i64 = sqlx::query("SELECT COUNT(*) AS n FROM board_lists WHERE space_id = ?1")
         .bind(space_id)
-        .fetch_one(&state.db)
+        .fetch_one(&mut *tx)
         .await?
         .get("n");
     if remaining <= 1 {
@@ -465,8 +523,11 @@ pub async fn remove_list(
 
     sqlx::query("DELETE FROM board_lists WHERE id = ?1")
         .bind(list_id)
-        .execute(&state.db)
+        .execute(&mut *tx)
         .await?;
+    // Deleting the last list makes the one before it "done".
+    settle(&mut tx, space_id).await?;
+    tx.commit().await?;
 
     state.backup.signal();
     let space_name = board_space(&state.db, space_id).await?;
@@ -479,17 +540,19 @@ pub async fn create_card(
     Path(list_id): Path<i64>,
     Json(input): Json<CardInput>,
 ) -> AppResult<Json<Card>> {
-    list_space(&state.db, list_id).await?;
+    let space_id = list_space(&state.db, list_id).await?;
     let title = input.title.trim();
     if title.is_empty() {
         return Err(AppError::BadRequest("a card needs a title".into()));
     }
+    linkable(&state.db, input.node_id).await?;
 
+    let mut tx = crate::db::begin_write(&state.db).await?;
     let position: i64 = sqlx::query(
         "SELECT COALESCE(MAX(position), -1) + 1 AS next FROM cards WHERE list_id = ?1",
     )
     .bind(list_id)
-    .fetch_one(&state.db)
+    .fetch_one(&mut *tx)
     .await?
     .get("next");
 
@@ -506,9 +569,12 @@ pub async fn create_card(
     .bind(input.due_at)
     .bind(position)
     .bind(now)
-    .fetch_one(&state.db)
+    .fetch_one(&mut *tx)
     .await?
     .get("id");
+    // A card made straight into the done list is finished as it lands.
+    settle(&mut tx, space_id).await?;
+    tx.commit().await?;
 
     state.backup.signal();
     card(&state.db, id).await.map(Json)
@@ -528,30 +594,50 @@ pub async fn update_card(
             return Err(AppError::BadRequest("a card needs a title".into()))
         }
         Some(title) => title.trim().to_string(),
-        None => current.title,
+        None => current.title.clone(),
     };
-    // `done` is a flag on the way in and a timestamp in the table: when it was
+    if let Some(node_id) = patch.node_id {
-    // finished is worth more than that it was, and the weekly digest needs it.
+        linkable(&state.db, node_id).await?;
-    let done_at = match patch.done {
+    }
-        Some(true) => current.done_at.or(Some(now)),
+
-        Some(false) => None,
+    let mut tx = crate::db::begin_write(&state.db).await?;
-        None => current.done_at,
+
+    // `done` is where the card stands, not a flag beside it: finishing a card
+    // moves it into the last list, reopening one moves it back to the first,
+    // and the stamp follows from the move as it does for any other.
+    if let Some(done) = patch.done.filter(|&done| done != current.done_at.is_some()) {
+        let space_id = list_space(&state.db, current.list_id).await?;
+        let lists: Vec<i64> = sqlx::query_scalar(
+            "SELECT id FROM board_lists WHERE space_id = ?1 ORDER BY position, id",
+        )
+        .bind(space_id)
+        .fetch_all(&mut *tx)
+        .await?;
+        let (Some(&first), Some(&last)) = (lists.first(), lists.last()) else {
+            return Err(AppError::NotFound);
         };
+        if lists.len() < 2 {
+            return Err(AppError::BadRequest(
+                "a board needs two lists for one to mean done".into(),
+            ));
+        }
+        let to = if done { last } else { first };
+        relocate(&mut tx, &current, to, last, None, now).await?;
+    }
 
     sqlx::query(
-        "UPDATE cards SET title = ?1, body = ?2, node_id = ?3, due_at = ?4, done_at = ?5,
+        "UPDATE cards SET title = ?1, body = ?2, node_id = ?3, due_at = ?4, updated_at = ?5
-                          updated_at = ?6
+         WHERE id = ?6",
-         WHERE id = ?7",
     )
     .bind(&title)
     .bind(patch.body.unwrap_or(current.body))
     .bind(patch.node_id.unwrap_or(current.node_id))
     .bind(patch.due_at.unwrap_or(current.due_at))
-    .bind(done_at)
     .bind(now)
     .bind(card_id)
-    .execute(&state.db)
+    .execute(&mut *tx)
     .await?;
+    tx.commit().await?;
 
     state.backup.signal();
     card(&state.db, card_id).await.map(Json)
@@ -595,57 +681,68 @@ pub async fn move_card(
         ));
     }
 
-    let mut tx = state.db.begin().await?;
+    let mut tx = crate::db::begin_write(&state.db).await?;
-
     let now = crate::now();
     let last = last_list(&mut *tx, to).await?.map_or(input.list_id, |(id, _)| id);
-    let done_at = done_after_move(current.list_id, input.list_id, last, current.done_at, now);
+    relocate(&mut tx, &current, input.list_id, last, input.position, now).await?;
+    tx.commit().await?;
+
+    state.backup.signal();
+    card(&state.db, card_id).await.map(Json)
+}
+
+/// Put a card on `to` at `position` (the end, if none), stamping or clearing
+/// its done time by where it lands, and close the ranks of both lists.
+async fn relocate(
+    db: &mut SqliteConnection,
+    current: &Card,
+    to: i64,
+    last: i64,
+    position: Option<i64>,
+    now: i64,
+) -> AppResult<()> {
+    let card_id = current.id;
+    let done_at = done_after_move(current.list_id, to, last, current.done_at, now);
 
     sqlx::query("UPDATE cards SET list_id = ?1, done_at = ?2, updated_at = ?3 WHERE id = ?4")
-        .bind(input.list_id)
+        .bind(to)
         .bind(done_at)
         .bind(now)
         .bind(card_id)
-        .execute(&mut *tx)
+        .execute(&mut *db)
         .await?;
 
-    let ids: Vec<i64> = sqlx::query("SELECT id FROM cards WHERE list_id = ?1 ORDER BY position, id")
+    let ids: Vec<i64> = sqlx::query_scalar("SELECT id FROM cards WHERE list_id = ?1 ORDER BY position, id")
-        .bind(input.list_id)
+        .bind(to)
-        .fetch_all(&mut *tx)
+        .fetch_all(&mut *db)
-        .await?
+        .await?;
-        .iter()
-        .map(|row| row.get("id"))
-        .collect();
 
-    let target = input.position.unwrap_or(i64::MAX).max(0) as usize;
+    let target = position.unwrap_or(i64::MAX).max(0) as usize;
     for (position, id) in reordered(&ids, card_id, target).iter().enumerate() {
         sqlx::query("UPDATE cards SET position = ?1 WHERE id = ?2")
             .bind(position as i64)
             .bind(id)
-            .execute(&mut *tx)
+            .execute(&mut *db)
             .await?;
     }
 
     // The list it came from keeps a hole where the card was. Harmless for
     // ordering, but it would grow with every move.
-    let left: Vec<i64> = sqlx::query("SELECT id FROM cards WHERE list_id = ?1 ORDER BY position, id")
+    if current.list_id != to {
+        let left: Vec<i64> =
+            sqlx::query_scalar("SELECT id FROM cards WHERE list_id = ?1 ORDER BY position, id")
                 .bind(current.list_id)
-        .fetch_all(&mut *tx)
+                .fetch_all(&mut *db)
-        .await?
+                .await?;
-        .iter()
-        .map(|row| row.get("id"))
-        .collect();
         for (position, id) in left.iter().enumerate() {
             sqlx::query("UPDATE cards SET position = ?1 WHERE id = ?2")
                 .bind(position as i64)
                 .bind(id)
-            .execute(&mut *tx)
+                .execute(&mut *db)
                 .await?;
         }
-
+    }
-    tx.commit().await?;
+    Ok(())
-    state.backup.signal();
-    card(&state.db, card_id).await.map(Json)
 }
 
 async fn card(db: &SqlitePool, id: i64) -> AppResult<Card> {
@@ -665,7 +762,7 @@ async fn card(db: &SqlitePool, id: i64) -> AppResult<Card> {
 
 #[cfg(test)]
 mod tests {
-    use super::{agenda_cards, done_after_move, reordered};
+    use super::{agenda_cards, card, done_after_move, relocate, reordered, settle};
 
     /// The agenda asks for what is due soon or in hand, and never for what is
     /// finished, parked in the inbox without a date, or due far off.
@@ -733,6 +830,70 @@ mod tests {
         );
     }
 
+    /// Whatever changes — a card moved, a list added after a lone one, the
+    /// last list deleted — a card is finished exactly when it is in the last.
+    #[tokio::test]
+    async fn done_is_where_a_card_stands() {
+        let db = sqlx::sqlite::SqlitePoolOptions::new()
+            .max_connections(1)
+            .connect("sqlite::memory:")
+            .await
+            .unwrap();
+        sqlx::migrate!("./migrations").run(&db).await.unwrap();
+        let space: i64 = sqlx::query_scalar(
+            "INSERT INTO nodes (parent_id, kind, name, slug, body, has_board, created_at, updated_at)
+             VALUES (NULL, 'space', 'job', 'job', '', 1, 0, 0) RETURNING id",
+        )
+        .fetch_one(&db)
+        .await
+        .unwrap();
+        let list = |position: i64| {
+            let db = db.clone();
+            async move {
+                sqlx::query_scalar::<_, i64>(
+                    "INSERT INTO board_lists (space_id, name, position, created_at)
+                     VALUES (?1, 'l', ?2, 0) RETURNING id",
+                )
+                .bind(space)
+                .bind(position)
+                .fetch_one(&db)
+                .await
+                .unwrap()
+            }
+        };
+        let (inbox, done) = (list(0).await, list(1).await);
+        let id: i64 = sqlx::query_scalar(
+            "INSERT INTO cards (list_id, title, position, created_at, updated_at)
+             VALUES (?1, 'c', 0, 0, 0) RETURNING id",
+        )
+        .bind(inbox)
+        .fetch_one(&db)
+        .await
+        .unwrap();
+        let done_at = |db: sqlx::SqlitePool| async move { card(&db, id).await.unwrap().done_at };
+
+        let current = card(&db, id).await.unwrap();
+        let mut conn = db.acquire().await.unwrap();
+        relocate(&mut conn, &current, done, done, None, 100).await.unwrap();
+        drop(conn);
+        assert_eq!(done_at(db.clone()).await, Some(100));
+
+        // A list added after the last one becomes "done", so the card that
+        // was finished in the old last list is open again.
+        let later = list(2).await;
+        settle(&mut db.acquire().await.unwrap(), space).await.unwrap();
+        assert_eq!(done_at(db.clone()).await, None);
+
+        // Deleting the new last list hands "done" back to the old one.
+        sqlx::query("DELETE FROM board_lists WHERE id = ?1")
+            .bind(later)
+            .execute(&db)
+            .await
+            .unwrap();
+        settle(&mut db.acquire().await.unwrap(), space).await.unwrap();
+        assert!(done_at(db.clone()).await.is_some());
+    }
+
     #[test]
     fn arriving_in_the_last_list_finishes_a_card_and_leaving_reopens_it() {
         let (inbox, doing, done) = (1, 2, 3);

src/routes/import.rs

@@ -404,8 +404,23 @@ fn unpack(
 
         let id = Uuid::new_v4().to_string();
         let target = uploads_dir.join(&id);
+        let partial = target.with_extension("part");
         ids.push(id.clone());
-        let size = std::io::copy(&mut capped, &mut std::fs::File::create(&target)?)?;
+        // Written aside and renamed into place whole, like any other upload,
+        // so a name the database may come to know never holds half a file.
+        let mut out = std::fs::File::create(&partial)?;
+        let copied = std::io::copy(&mut capped, &mut out).and_then(|size| {
+            out.sync_all()?;
+            Ok(size)
+        });
+        let size = match copied {
+            Ok(size) => size,
+            Err(e) => {
+                let _ = std::fs::remove_file(&partial);
+                return Err(e.into());
+            }
+        };
+        std::fs::rename(&partial, &target)?;
         unpacked += size;
         if unpacked > limit {
             return Err(unpacked_too_large(limit));

src/routes/media.rs

@@ -9,7 +9,7 @@ use axum::{
 };
 use regex::Regex;
 use serde::{Deserialize, Serialize};
-use sqlx::{Row, SqlitePool};
+use sqlx::{Connection, Row, SqliteConnection, SqlitePool};
 use tokio::io::AsyncWriteExt;
 use tokio_util::io::ReaderStream;
 use uuid::Uuid;
@@ -74,6 +74,9 @@ pub struct Media {
     pub encrypted: bool,
     /// Every document that embeds this file, which may be none or several.
     pub node_ids: Vec<i64>,
+    /// The file node it is, when it sits in the tree: deleted from there, not
+    /// from the media list.
+    pub file_node: Option<i64>,
 }
 
 fn row_to_media(row: &sqlx::sqlite::SqliteRow) -> Media {
@@ -96,6 +99,7 @@ fn row_to_media(row: &sqlx::sqlite::SqliteRow) -> Media {
         created_at: row.get("created_at"),
         encrypted: row.try_get("encrypted").unwrap_or(false),
         node_ids,
+        file_node: row.try_get("file_node").ok().flatten(),
     }
 }
 
@@ -164,6 +168,9 @@ pub(crate) async fn store_field(
             file.write_all(&sealer.finish()?).await?;
         }
         file.flush().await?;
+        // On disk before the rename, and so before any row names it: after a
+        // power cut the database must not know a file whose bytes never landed.
+        file.sync_all().await?;
         Ok(size)
     }
     .await;
@@ -253,7 +260,7 @@ pub async fn upload_into(
     Query(query): Query<UploadQuery>,
     mut multipart: Multipart,
 ) -> AppResult<Json<Vec<nodes::Node>>> {
-    nodes::container(&state.db, parent).await?;
+    nodes::container(&mut *state.db.acquire().await?, parent).await?;
     let key = if query.encrypt {
         Some(state.vault.key().ok_or(AppError::Locked)?)
     } else {
@@ -315,9 +322,13 @@ async fn insert_file(
     size: u64,
     encrypted: bool,
 ) -> AppResult<i64> {
-    let slug = nodes::unique_slug(db, Some(parent), &nodes::slugify(filename), None).await?;
+    let mut tx = crate::db::begin_write(db).await?;
+    // Checked inside the transaction that takes it, so the slug is still
+    // free when the node is written; the folder too, which may have gone
+    // while the upload was streaming in.
+    nodes::container(&mut tx, parent).await?;
+    let slug = nodes::unique_slug(&mut tx, Some(parent), &nodes::slugify(filename), None).await?;
     let now = crate::now();
-    let mut tx = db.begin().await?;
 
     sqlx::query(
         "INSERT INTO media (id, filename, mime, size, created_at, encrypted)
@@ -359,8 +370,10 @@ async fn insert_file(
 pub async fn list(_: Session, State(state): State<AppState>) -> AppResult<Json<Vec<Media>>> {
     let rows = sqlx::query(
         "SELECT m.id, m.filename, m.mime, m.size, m.created_at, m.encrypted,
+                (SELECT f.id FROM nodes f WHERE f.media_id = m.id) AS file_node,
                 (SELECT group_concat(nm.node_id)
-                 FROM node_media nm WHERE nm.media_id = m.id) AS node_ids
+                 FROM node_media nm JOIN nodes d ON d.id = nm.node_id
+                 WHERE nm.media_id = m.id AND d.kind <> 'file') AS node_ids
          FROM media m ORDER BY m.created_at DESC LIMIT 500",
     )
     .fetch_all(&state.db)
@@ -486,9 +499,22 @@ pub(crate) fn canonical_id(id: &str) -> Option<String> {
 
 pub async fn delete_media(state: &AppState, id: &str) -> AppResult<()> {
     let id = canonical_id(id).ok_or(AppError::NotFound)?;
+    let mut tx = crate::db::begin_write(&state.db).await?;
+    // A file in the tree is its node's to delete: going round it here would
+    // take the node out of its folder without the tree having been asked.
+    let in_tree = sqlx::query("SELECT 1 FROM nodes WHERE media_id = ?1")
+        .bind(&id)
+        .fetch_optional(&mut *tx)
+        .await?
+        .is_some();
+    if in_tree {
+        return Err(AppError::BadRequest(
+            "that is a file in the tree — delete it from its folder".into(),
+        ));
+    }
     let deleted = sqlx::query("DELETE FROM media WHERE id = ?1")
         .bind(&id)
-        .execute(&state.db)
+        .execute(&mut *tx)
         .await?
         .rows_affected();
     // No row, no file of ours: nothing on disk is removed on the word of a
@@ -496,26 +522,38 @@ pub async fn delete_media(state: &AppState, id: &str) -> AppResult<()> {
     if deleted == 0 {
         return Err(AppError::NotFound);
     }
-    let path = state.config.uploads_dir().join(&id);
+    tx.commit().await?;
-    if let Err(e) = tokio::fs::remove_file(&path).await {
+    remove_upload(&state.config.uploads_dir(), &id).await;
-        if e.kind() != std::io::ErrorKind::NotFound {
-            return Err(e.into());
-        }
-    }
     Ok(())
 }
 
+/// The bytes of an upload whose row is already gone. Called after the
+/// commit, never before: a file a crash leaves behind is swept up later as an
+/// orphan, where a row whose file had gone first would be a broken link.
+pub async fn remove_upload(uploads: &std::path::Path, id: &str) {
+    match tokio::fs::remove_file(uploads.join(id)).await {
+        Ok(()) => {}
+        Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
+        Err(e) => tracing::warn!("could not remove upload {id}: {e}; the sweep will retry"),
+    }
+}
+
 /// Attach every media file the body embeds to this document, and detach the
 /// ones it no longer mentions so they stop being reachable through its share
 /// link. Other documents keep whatever they embed; this rewrites one node's rows.
-pub async fn link_to_node(db: &SqlitePool, node_id: i64, body: &str) -> AppResult<()> {
+pub async fn link_to_node(
+    db: &mut SqliteConnection,
+    node_id: i64,
+    body: &str,
+) -> AppResult<()> {
     let referenced: BTreeSet<String> = MEDIA_REF
         .captures_iter(body)
         .map(|c| c[1].to_lowercase())
         .collect();
 
-    // One transaction, so a document is never momentarily attached to nothing —
+    // One transaction — a savepoint inside the caller's, when there is one —
-    // which would blank its images for anyone reading its share link.
+    // so a document is never momentarily attached to nothing, which would
+    // blank its images for anyone reading its share link.
     let mut tx = db.begin().await?;
 
     sqlx::query("DELETE FROM node_media WHERE node_id = ?1")
@@ -544,7 +582,10 @@ pub async fn link_to_node(db: &SqlitePool, node_id: i64, body: &str) -> AppResul
 /// that become unreachable once those nodes are gone. A whole subtree is asked
 /// about at once, because deleting a folder deletes everything under it and a
 /// file shared between two of its documents is still exclusive to the subtree.
-pub async fn exclusive_media(db: &SqlitePool, node_ids: &[i64]) -> AppResult<Vec<String>> {
+pub async fn exclusive_media(
+    db: &mut SqliteConnection,
+    node_ids: &[i64],
+) -> AppResult<Vec<String>> {
     if node_ids.is_empty() {
         return Ok(Vec::new());
     }
@@ -566,7 +607,7 @@ pub async fn exclusive_media(db: &SqlitePool, node_ids: &[i64]) -> AppResult<Vec
                WHERE other.media_id = nm.media_id AND other.node_id NOT IN ({list})
            )"
     ))
-    .fetch_all(db)
+    .fetch_all(&mut *db)
     .await?;
 
     Ok(rows.iter().map(|row| row.get("id")).collect())

src/routes/nodes.rs

@@ -5,7 +5,7 @@ use axum::{
     Json,
 };
 use serde::{Deserialize, Serialize};
-use sqlx::{Row, SqlitePool};
+use sqlx::{Row, SqliteConnection, SqlitePool};
 
 use crate::{
     auth::Session,
@@ -23,6 +23,11 @@ pub const FOLDER: &str = "folder";
 pub const DOCUMENT: &str = "document";
 pub const FILE: &str = "file";
 
+/// The `role` of the one space that is the diary: a journal, with no board,
+/// whose empty days the reminder mail is about. Found by role rather than by
+/// slug, so it stays the diary under any name.
+pub const DIARY: &str = "diary";
+
 /// Whether a node of this kind can hold other nodes.
 pub fn is_container(kind: &str) -> bool {
     kind == SPACE || kind == FOLDER
@@ -49,6 +54,8 @@ pub struct NodeSummary {
     pub excerpt: String,
     pub position: i64,
     pub has_board: bool,
+    /// Spaces the code has to tell apart: `diary`, or nothing.
+    pub role: Option<String>,
     pub child_count: i64,
     pub created_at: i64,
     pub updated_at: i64,
@@ -71,6 +78,7 @@ pub struct Node {
     pub excerpt: String,
     pub position: i64,
     pub has_board: bool,
+    pub role: Option<String>,
     pub child_count: i64,
     pub created_at: i64,
     pub updated_at: i64,
@@ -156,7 +164,7 @@ const HEAD_CHARS: usize = 1024;
 const EXCERPT_CHARS: usize = 180;
 
 /// The columns every list query selects, so a row always converts the same way.
-const SUMMARY_COLUMNS: &str = "n.id, n.parent_id, n.kind, n.name, n.slug, n.position, n.has_board,
+const SUMMARY_COLUMNS: &str = "n.id, n.parent_id, n.kind, n.name, n.slug, n.position, n.has_board, n.role,
      n.created_at, n.updated_at, n.share_token, n.media_id,
      (SELECT COUNT(*) FROM nodes c WHERE c.parent_id = n.id) AS child_count,
      (SELECT m.mime FROM media m WHERE m.id = n.media_id) AS file_mime,
@@ -254,7 +262,7 @@ pub fn next_slug(taken: &BTreeSet<String>, base: &str) -> AppResult<String> {
 /// The slugs already spoken for under `parent_id`. `exclude` is the node being
 /// renamed, which must not collide with itself.
 pub async fn sibling_slugs(
-    db: &SqlitePool,
+    db: &mut SqliteConnection,
     parent_id: Option<i64>,
     exclude: Option<i64>,
 ) -> AppResult<BTreeSet<String>> {
@@ -264,7 +272,7 @@ pub async fn sibling_slugs(
     )
     .bind(parent_id)
     .bind(exclude)
-    .fetch_all(db)
+    .fetch_all(&mut *db)
     .await?;
 
     Ok(rows.iter().map(|row| row.get("slug")).collect())
@@ -272,7 +280,7 @@ pub async fn sibling_slugs(
 
 /// `base`, or `base-2`, `base-3`… until no sibling holds it.
 pub async fn unique_slug(
-    db: &SqlitePool,
+    db: &mut SqliteConnection,
     parent_id: Option<i64>,
     base: &str,
     exclude: Option<i64>,
@@ -300,6 +308,7 @@ fn row_to_summary(row: &sqlx::sqlite::SqliteRow, head: &str) -> NodeSummary {
         excerpt: excerpt(head),
         position: row.get("position"),
         has_board: row.get::<i64, _>("has_board") != 0,
+        role: row.get("role"),
         child_count: row.get("child_count"),
         created_at: row.get("created_at"),
         updated_at: row.get("updated_at"),
@@ -390,10 +399,11 @@ pub async fn create_space(
 /// A new space, last in the switcher. Shared by the API and by `mkdir` at the
 /// root of the SFTP mount, which is why it takes a pool and not the state.
 pub async fn insert_space(db: &SqlitePool, name: &str, has_board: bool) -> AppResult<i64> {
-    let slug = unique_slug(db, None, &slugify(name), None).await?;
+    let mut tx = crate::db::begin_write(db).await?;
+    let slug = unique_slug(&mut tx, None, &slugify(name), None).await?;
     let now = crate::now();
     let position: i64 = sqlx::query("SELECT COALESCE(MAX(position), -1) + 1 AS next FROM nodes WHERE parent_id IS NULL")
-        .fetch_one(db)
+        .fetch_one(&mut *tx)
         .await?
         .get("next");
 
@@ -407,10 +417,11 @@ pub async fn insert_space(db: &SqlitePool, name: &str, has_board: bool) -> AppRe
     .bind(position)
     .bind(i64::from(has_board))
     .bind(now)
-    .fetch_one(db)
+    .fetch_one(&mut *tx)
     .await?
     .get("id");
 
+    tx.commit().await?;
     Ok(id)
 }
 
@@ -526,17 +537,12 @@ pub async fn create(
         )));
     }
 
-    container(&state.db, input.parent_id).await?;
+    let mut tx = crate::db::begin_write(&state.db).await?;
+    container(&mut tx, input.parent_id).await?;
 
     let now = crate::now();
     let created_at = input.created_at.unwrap_or(now);
-    let slug = unique_slug(
+    let slug = unique_slug(&mut tx, Some(input.parent_id), &slugify(&input.name), None).await?;
-        &state.db,
-        Some(input.parent_id),
-        &slugify(&input.name),
-        None,
-    )
-    .await?;
 
     let id: i64 = sqlx::query(
         "INSERT INTO nodes (parent_id, kind, name, slug, body, created_at, updated_at)
@@ -550,11 +556,12 @@ pub async fn create(
     .bind(&input.body)
     .bind(created_at)
     .bind(now)
-    .fetch_one(&state.db)
+    .fetch_one(&mut *tx)
     .await?
     .get("id");
 
-    media::link_to_node(&state.db, id, &input.body).await?;
+    media::link_to_node(&mut tx, id, &input.body).await?;
+    tx.commit().await?;
     state.backup.signal();
     load(&state, id).await.map(Json)
 }
@@ -565,9 +572,13 @@ pub async fn update(
     Path(id): Path<i64>,
     Json(input): Json<UpdateInput>,
 ) -> AppResult<Json<Node>> {
+    // One transaction from the first read to the last write: the slug chosen
+    // here is still free when it is written, and a file's name and its
+    // download name never disagree.
+    let mut tx = crate::db::begin_write(&state.db).await?;
     let row = sqlx::query("SELECT parent_id, kind, name, slug FROM nodes WHERE id = ?1")
         .bind(id)
-        .fetch_optional(&state.db)
+        .fetch_optional(&mut *tx)
         .await?
         .ok_or(AppError::NotFound)?;
 
@@ -582,7 +593,7 @@ pub async fn update(
     let slug = if name == previous_name {
         row.get::<String, _>("slug")
     } else {
-        unique_slug(&state.db, parent_id, &slugify(name), Some(id)).await?
+        unique_slug(&mut tx, parent_id, &slugify(name), Some(id)).await?
     };
 
     if input.has_board.is_some() && kind != SPACE {
@@ -596,11 +607,11 @@ pub async fn update(
         sqlx::query("UPDATE media SET filename = ?1 WHERE id = (SELECT media_id FROM nodes WHERE id = ?2)")
             .bind(name)
             .bind(id)
-            .execute(&state.db)
+            .execute(&mut *tx)
             .await?;
     }
 
-    let affected = sqlx::query(
+    sqlx::query(
         "UPDATE nodes
          SET name = ?1, body = ?2, slug = ?3, updated_at = ?4,
              created_at = COALESCE(?5, created_at),
@@ -614,17 +625,13 @@ pub async fn update(
     .bind(input.created_at)
     .bind(input.has_board.map(i64::from))
     .bind(id)
-    .execute(&state.db)
+    .execute(&mut *tx)
-    .await?
+    .await?;
-    .rows_affected();
-
-    if affected == 0 {
-        return Err(AppError::NotFound);
-    }
 
     if kind != FILE {
-        media::link_to_node(&state.db, id, body).await?;
+        media::link_to_node(&mut tx, id, body).await?;
     }
+    tx.commit().await?;
     state.backup.signal();
     load(&state, id).await.map(Json)
 }
@@ -634,34 +641,55 @@ pub async fn remove(
     State(state): State<AppState>,
     Path(id): Path<i64>,
 ) -> AppResult<Json<serde_json::Value>> {
-    // Everything under a folder goes with it, so the files to clean up are
+    let removed = delete_subtree(&state.db, &state.config.uploads_dir(), id).await?;
-    // those exclusive to the whole subtree. Collected before the delete, which
-    // takes the node_media rows with it.
-    let subtree = descendants(&state.db, id).await?;
-    if subtree.is_empty() {
-        return Err(AppError::NotFound);
-    }
-    let orphans = media::exclusive_media(&state.db, &subtree).await?;
-
-    let affected = sqlx::query("DELETE FROM nodes WHERE id = ?1")
-        .bind(id)
-        .execute(&state.db)
-        .await?
-        .rows_affected();
-
-    if affected == 0 {
-        return Err(AppError::NotFound);
-    }
-
-    for media_id in orphans {
-        match media::delete_media(&state, &media_id).await {
-            Ok(()) | Err(AppError::NotFound) => {}
-            Err(e) => return Err(e),
-        }
-    }
-
     state.backup.signal();
-    Ok(Json(serde_json::json!({ "ok": true, "removed": subtree.len() })))
+    Ok(Json(serde_json::json!({ "ok": true, "removed": removed })))
+}
+
+/// Delete a node, everything beneath it, and the files nothing outside it
+/// embeds. The one way a node is deleted, so a delete through the SFTP mount
+/// cleans up exactly what one through the API does. Returns how many nodes
+/// went.
+pub async fn delete_subtree(
+    db: &SqlitePool,
+    uploads: &std::path::Path,
+    id: i64,
+) -> AppResult<usize> {
+    let mut tx = crate::db::begin_write(db).await?;
+
+    let role: Option<String> = sqlx::query_scalar("SELECT role FROM nodes WHERE id = ?1")
+        .bind(id)
+        .fetch_optional(&mut *tx)
+        .await?
+        .ok_or(AppError::NotFound)?;
+    if role.as_deref() == Some(DIARY) {
+        return Err(AppError::BadRequest(
+            "the diary space stays — empty it instead, or rename it".into(),
+        ));
+    }
+
+    // Collected before the delete, which takes the node_media rows with it.
+    // Everything under a folder goes with it, so the files to clean up are
+    // those exclusive to the whole subtree.
+    let subtree = descendants(&mut tx, id).await?;
+    let orphans = media::exclusive_media(&mut tx, &subtree).await?;
+
+    sqlx::query("DELETE FROM nodes WHERE id = ?1")
+        .bind(id)
+        .execute(&mut *tx)
+        .await?;
+    for media_id in &orphans {
+        sqlx::query("DELETE FROM media WHERE id = ?1")
+            .bind(media_id)
+            .execute(&mut *tx)
+            .await?;
+    }
+    tx.commit().await?;
+
+    for media_id in &orphans {
+        media::remove_upload(uploads, media_id).await;
+    }
+    Ok(subtree.len())
 }
 
 /// Reparent a node, and optionally place it among its new siblings.
@@ -675,31 +703,33 @@ pub async fn move_node(
         return Err(AppError::BadRequest("a node cannot contain itself".into()));
     }
 
-    container(&state.db, input.parent_id).await?;
+    let mut tx = crate::db::begin_write(&state.db).await?;
+
+    let row = sqlx::query("SELECT kind, name FROM nodes WHERE id = ?1")
+        .bind(id)
+        .fetch_optional(&mut *tx)
+        .await?
+        .ok_or(AppError::NotFound)?;
+    let kind: String = row.get("kind");
+    let name: String = row.get("name");
+    // A space is a root: under a folder it would drop out of the switcher
+    // while its board and its cards carried on as if nothing had happened.
+    if kind == SPACE {
+        return Err(AppError::BadRequest("a space cannot be moved into anything".into()));
+    }
+
+    container(&mut tx, input.parent_id).await?;
 
     // Moving a folder into its own subtree would cut that subtree loose from
     // every root, where nothing could reach it again.
-    if descendants(&state.db, id).await?.contains(&input.parent_id) {
+    if descendants(&mut tx, id).await?.contains(&input.parent_id) {
         return Err(AppError::BadRequest(
             "a folder cannot be moved into itself".into(),
         ));
     }
 
-    let name: String = sqlx::query("SELECT name FROM nodes WHERE id = ?1")
-        .bind(id)
-        .fetch_optional(&state.db)
-        .await?
-        .ok_or(AppError::NotFound)?
-        .get("name");
-
     // The slug has to be free where it lands, not where it came from.
-    let slug = unique_slug(
+    let slug = unique_slug(&mut tx, Some(input.parent_id), &slugify(&name), Some(id)).await?;
-        &state.db,
-        Some(input.parent_id),
-        &slugify(&name),
-        Some(id),
-    )
-    .await?;
 
     sqlx::query(
         "UPDATE nodes
@@ -711,18 +741,22 @@ pub async fn move_node(
     .bind(input.position)
     .bind(crate::now())
     .bind(id)
-    .execute(&state.db)
+    .execute(&mut *tx)
     .await?;
+    tx.commit().await?;
 
     state.backup.signal();
     load(&state, id).await.map(Json)
 }
 
 /// Refuse anything but a space or a folder as the place a node goes into.
-pub async fn container(db: &SqlitePool, id: i64) -> AppResult<()> {
+pub async fn container(
+    db: &mut SqliteConnection,
+    id: i64,
+) -> AppResult<()> {
     let kind: String = sqlx::query("SELECT kind FROM nodes WHERE id = ?1")
         .bind(id)
-        .fetch_optional(db)
+        .fetch_optional(&mut *db)
         .await?
         .ok_or(AppError::NotFound)?
         .get("kind");
@@ -735,7 +769,10 @@ pub async fn container(db: &SqlitePool, id: i64) -> AppResult<()> {
 }
 
 /// A node and everything beneath it, the node itself first.
-pub async fn descendants(db: &SqlitePool, id: i64) -> AppResult<Vec<i64>> {
+pub async fn descendants(
+    db: &mut SqliteConnection,
+    id: i64,
+) -> AppResult<Vec<i64>> {
     let rows = sqlx::query(
         "WITH RECURSIVE sub (id) AS (
              SELECT id FROM nodes WHERE id = ?1
@@ -745,7 +782,7 @@ pub async fn descendants(db: &SqlitePool, id: i64) -> AppResult<Vec<i64>> {
          SELECT id FROM sub",
     )
     .bind(id)
-    .fetch_all(db)
+    .fetch_all(&mut *db)
     .await?;
 
     Ok(rows.iter().map(|row| row.get("id")).collect())
@@ -799,6 +836,7 @@ pub async fn load(state: &AppState, id: i64) -> AppResult<Node> {
         excerpt: summary.excerpt,
         position: summary.position,
         has_board: summary.has_board,
+        role: summary.role,
         child_count: summary.child_count,
         created_at: summary.created_at,
         updated_at: summary.updated_at,
@@ -814,8 +852,89 @@ mod tests {
     use std::collections::BTreeSet;
 
     use super::{
-        excerpt, fts_query, next_slug, recent_documents, slugify, EXCERPT_CHARS, HEAD_CHARS,
+        delete_subtree, excerpt, fts_query, next_slug, recent_documents, slugify, EXCERPT_CHARS,
+        HEAD_CHARS,
     };
+    use crate::error::AppError;
+
+    /// A delete takes the subtree and the files only it embedded — from the
+    /// table and from the disk — and leaves what something else still embeds.
+    /// The diary, found by its role whatever it is called, is not deleted.
+    #[tokio::test]
+    async fn a_delete_takes_what_only_it_held() {
+        let db = sqlx::sqlite::SqlitePoolOptions::new()
+            .max_connections(2)
+            .connect("sqlite::memory:")
+            .await
+            .unwrap();
+        sqlx::migrate!("./migrations").run(&db).await.unwrap();
+        let dir = tempfile::tempdir().unwrap();
+
+        let diary: i64 = sqlx::query_scalar("SELECT id FROM nodes WHERE role = 'diary'")
+            .fetch_one(&db)
+            .await
+            .unwrap();
+        sqlx::query("UPDATE nodes SET name = 'journal', slug = 'journal' WHERE id = ?1")
+            .bind(diary)
+            .execute(&db)
+            .await
+            .unwrap();
+        assert!(matches!(
+            delete_subtree(&db, dir.path(), diary).await,
+            Err(AppError::BadRequest(_))
+        ));
+
+        let node = |parent: i64, kind: &'static str, slug: &'static str| {
+            let db = db.clone();
+            async move {
+                sqlx::query_scalar::<_, i64>(
+                    "INSERT INTO nodes (parent_id, kind, name, slug, created_at, updated_at)
+                     VALUES (?1, ?2, ?3, ?3, 0, 0) RETURNING id",
+                )
+                .bind(parent)
+                .bind(kind)
+                .bind(slug)
+                .fetch_one(&db)
+                .await
+                .unwrap()
+            }
+        };
+        let folder = node(diary, "folder", "trip").await;
+        let inside = node(folder, "document", "day-1").await;
+        let outside = node(diary, "document", "elsewhere").await;
+
+        let only = "11111111-1111-4111-8111-111111111111";
+        let shared = "22222222-2222-4222-8222-222222222222";
+        for (id, nodes) in [(only, vec![inside]), (shared, vec![inside, outside])] {
+            sqlx::query("INSERT INTO media (id, filename, mime, size, created_at) VALUES (?1, 'x', 'image/png', 1, 0)")
+                .bind(id)
+                .execute(&db)
+                .await
+                .unwrap();
+            std::fs::write(dir.path().join(id), b"x").unwrap();
+            for node in nodes {
+                sqlx::query("INSERT INTO node_media (node_id, media_id) VALUES (?1, ?2)")
+                    .bind(node)
+                    .bind(id)
+                    .execute(&db)
+                    .await
+                    .unwrap();
+            }
+        }
+
+        assert_eq!(delete_subtree(&db, dir.path(), folder).await.unwrap(), 2);
+        let media: Vec<String> = sqlx::query_scalar("SELECT id FROM media")
+            .fetch_all(&db)
+            .await
+            .unwrap();
+        assert_eq!(media, [shared]);
+        assert!(!dir.path().join(only).exists());
+        assert!(dir.path().join(shared).exists());
+        assert!(matches!(
+            delete_subtree(&db, dir.path(), folder).await,
+            Err(AppError::NotFound)
+        ));
+    }
 
     /// The overview reaches into folders, newest edit first, and stops at the
     /// limit — and never lists a folder as if it were a document.

src/routes/seal.rs

@@ -60,7 +60,7 @@ struct Target {
 async fn reseal(state: &AppState, id: i64, seal: bool) -> AppResult<Resealed> {
     let key = state.vault.key().ok_or(AppError::Locked)?;
 
-    let subtree = nodes::descendants(&state.db, id).await?;
+    let subtree = nodes::descendants(&mut *state.db.acquire().await?, id).await?;
     if subtree.is_empty() {
         return Err(AppError::NotFound);
     }

src/sftp/fs.rs

@@ -19,7 +19,7 @@ use russh_sftp::protocol::{
 };
 use sqlx::{Row, SqlitePool};
 
-use crate::{backup::Backup, routes::media, routes::nodes, state::AppState};
+use crate::{backup::Backup, error::AppError, routes::media, routes::nodes, state::AppState};
 
 type SftpResult<T> = Result<T, StatusCode>;
 
@@ -175,6 +175,8 @@ enum OpenHandle {
 
 pub struct SpaceFs {
     db: SqlitePool,
+    /// Where the bytes of uploads live, so a delete here removes them too.
+    uploads: std::path::PathBuf,
     /// `None` only in tests, where there is nothing to mirror to.
     backup: Option<Arc<Backup>>,
     handles: HashMap<String, OpenHandle>,
@@ -184,12 +186,13 @@ pub struct SpaceFs {
 
 impl SpaceFs {
     pub fn new(state: AppState) -> Self {
-        Self::with_db(state.db, Some(state.backup))
+        Self::with_db(state.db, state.config.uploads_dir(), Some(state.backup))
     }
 
-    fn with_db(db: SqlitePool, backup: Option<Arc<Backup>>) -> Self {
+    fn with_db(db: SqlitePool, uploads: std::path::PathBuf, backup: Option<Arc<Backup>>) -> Self {
         Self {
             db,
+            uploads,
             backup,
             handles: HashMap::new(),
             next_handle: 0,
@@ -330,7 +333,7 @@ impl SpaceFs {
     /// An empty document under `parent`, named after the file it was created
     /// as until a heading says otherwise.
     async fn create_doc(&self, parent: i64, stem: &str) -> SftpResult<i64> {
-        let slug = nodes::unique_slug(&self.db, Some(parent), &nodes::slugify(stem), None)
+        let slug = nodes::unique_slug(&mut *self.conn().await?, Some(parent), &nodes::slugify(stem), None)
             .await
             .map_err(|_| StatusCode::Failure)?;
         let id: i64 = sqlx::query(
@@ -364,7 +367,7 @@ impl SpaceFs {
         .execute(&self.db)
         .await
         .map_err(|_| StatusCode::Failure)?;
-        media::link_to_node(&self.db, id, &body)
+        media::link_to_node(&mut *self.conn().await?, id, &body)
             .await
             .map_err(|_| StatusCode::Failure)?;
         self.changed();
@@ -372,15 +375,21 @@ impl SpaceFs {
     }
 
     async fn delete_node(&self, id: i64) -> SftpResult<()> {
-        sqlx::query("DELETE FROM nodes WHERE id = ?1")
+        nodes::delete_subtree(&self.db, &self.uploads, id)
-            .bind(id)
-            .execute(&self.db)
             .await
-            .map_err(|_| StatusCode::Failure)?;
+            .map_err(|e| match e {
+                AppError::NotFound => StatusCode::NoSuchFile,
+                AppError::BadRequest(_) => StatusCode::PermissionDenied,
+                _ => StatusCode::Failure,
+            })?;
         self.changed();
         Ok(())
     }
 
+    async fn conn(&self) -> SftpResult<sqlx::pool::PoolConnection<sqlx::Sqlite>> {
+        self.db.acquire().await.map_err(|_| StatusCode::Failure)
+    }
+
     async fn listing(&self, path: &str) -> SftpResult<Vec<File>> {
         let segs = segments(path);
         let parent = self.resolve_dir(&segs).await?;
@@ -699,7 +708,7 @@ impl russh_sftp::server::Handler for SpaceFs {
             self.changed();
             return Ok(ok_status(id));
         };
-        let slug = nodes::unique_slug(&self.db, Some(parent), &nodes::slugify(name), None)
+        let slug = nodes::unique_slug(&mut *self.conn().await?, Some(parent), &nodes::slugify(name), None)
             .await
             .map_err(|_| StatusCode::Failure)?;
         sqlx::query(
@@ -768,7 +777,7 @@ impl russh_sftp::server::Handler for SpaceFs {
                         return Err(StatusCode::PermissionDenied)
                     }
                     (_, Some(parent)) => {
-                        let below = nodes::descendants(&self.db, node)
+                        let below = nodes::descendants(&mut *self.conn().await?, node)
                             .await
                             .map_err(|_| StatusCode::Failure)?;
                         if below.contains(&parent) {
@@ -777,7 +786,7 @@ impl russh_sftp::server::Handler for SpaceFs {
                     }
                 }
                 let new_slug =
-                    nodes::unique_slug(&self.db, new_parent, &nodes::slugify(new_leaf), Some(node))
+                    nodes::unique_slug(&mut *self.conn().await?, new_parent, &nodes::slugify(new_leaf), Some(node))
                         .await
                         .map_err(|_| StatusCode::Failure)?;
                 // A move that keeps the name keeps the title too.
@@ -812,7 +821,7 @@ impl russh_sftp::server::Handler for SpaceFs {
                     }
                     (Some(stem), _) => {
                         let new_slug = nodes::unique_slug(
-                            &self.db,
+                            &mut *self.conn().await?,
                             Some(new_parent),
                             &nodes::slugify(stem),
                             Some(doc.id),
@@ -942,7 +951,7 @@ mod tests {
             .await
             .expect("an in-memory database");
         sqlx::migrate!("./migrations").run(&db).await.expect("migrations apply");
-        SpaceFs::with_db(db, None)
+        SpaceFs::with_db(db, std::env::temp_dir().join("narl-sftp-test-uploads"), None)
     }
 
     const CREATE: OpenFlags = OpenFlags::WRITE.union(OpenFlags::CREATE).union(OpenFlags::TRUNCATE);

src/sweep.rs

@@ -0,0 +1,187 @@
+//! Housekeeping for `uploads/`: what a crash, a killed upload or a forgotten
+//! embed leaves behind. Nothing else ever deletes these, so without this they
+//! would only grow — on the disk and in every backup of it.
+
+use std::{collections::HashSet, path::Path, time::Duration};
+
+use sqlx::SqlitePool;
+
+use crate::routes::media;
+
+/// A file this much younger may still be on its way in: an upload streaming
+/// to `.part`, an import whose rows are about to be written.
+const IN_FLIGHT: Duration = Duration::from_secs(60 * 60);
+
+/// How long an upload that nothing embeds is kept. Long enough to paste an
+/// image, delete the line by mistake, and put it back days later.
+const UNREFERENCED_DAYS: i64 = 7;
+
+const EVERY: Duration = Duration::from_secs(24 * 60 * 60);
+
+pub fn spawn(db: SqlitePool, uploads: std::path::PathBuf) {
+    tokio::spawn(async move {
+        let mut ticker = tokio::time::interval(EVERY);
+        loop {
+            ticker.tick().await;
+            match run(&db, &uploads).await {
+                Ok(swept) if swept.any() => tracing::info!(
+                    "swept uploads: {} partial, {} without a row, {} that nothing embeds",
+                    swept.partial,
+                    swept.stray,
+                    swept.unreferenced
+                ),
+                Ok(_) => {}
+                Err(e) => tracing::warn!("could not sweep uploads: {e:#}"),
+            }
+        }
+    });
+}
+
+#[derive(Debug, Default, PartialEq)]
+pub struct Swept {
+    pub partial: usize,
+    pub stray: usize,
+    pub unreferenced: usize,
+}
+
+impl Swept {
+    fn any(&self) -> bool {
+        self.partial + self.stray + self.unreferenced > 0
+    }
+}
+
+pub async fn run(db: &SqlitePool, uploads: &Path) -> anyhow::Result<Swept> {
+    let mut swept = Swept::default();
+
+    // Rows first, so the files they leave are strays by the time the
+    // directory is walked, and go in the same pass.
+    let cutoff = crate::now() - UNREFERENCED_DAYS * 86_400;
+    let unreferenced: Vec<String> = sqlx::query_scalar(
+        "SELECT m.id FROM media m
+         WHERE m.created_at < ?1
+           AND NOT EXISTS (SELECT 1 FROM node_media nm WHERE nm.media_id = m.id)
+           AND NOT EXISTS (SELECT 1 FROM nodes n WHERE n.media_id = m.id)
+           -- Belt and braces: a body or a card that names the file keeps it,
+           -- whatever the link table says.
+           AND NOT EXISTS (SELECT 1 FROM nodes n WHERE instr(n.body, m.id) > 0)
+           AND NOT EXISTS (SELECT 1 FROM cards c WHERE instr(c.body, m.id) > 0)",
+    )
+    .bind(cutoff)
+    .fetch_all(db)
+    .await?;
+    for id in &unreferenced {
+        sqlx::query("DELETE FROM media WHERE id = ?1")
+            .bind(id)
+            .execute(db)
+            .await?;
+    }
+    swept.unreferenced = unreferenced.len();
+
+    let known: HashSet<String> = sqlx::query_scalar("SELECT id FROM media")
+        .fetch_all(db)
+        .await?
+        .into_iter()
+        .collect();
+
+    let mut entries = tokio::fs::read_dir(uploads).await?;
+    while let Some(entry) = entries.next_entry().await? {
+        let name = entry.file_name().to_string_lossy().into_owned();
+        let meta = entry.metadata().await?;
+        if !meta.is_file() {
+            continue;
+        }
+        let age = meta.modified()?.elapsed().unwrap_or_default();
+        if age < IN_FLIGHT {
+            continue;
+        }
+        let kind = if name.ends_with(".part") || name.starts_with(".import-") {
+            &mut swept.partial
+        } else if media::canonical_id(&name).as_deref() == Some(name.as_str()) && !known.contains(&name) {
+            &mut swept.stray
+        } else {
+            // Not ours to judge: anything that is not a uuid or a leftover of
+            // one was put there by someone else.
+            continue;
+        };
+        match tokio::fs::remove_file(entry.path()).await {
+            Ok(()) => *kind += 1,
+            Err(e) => tracing::warn!("could not remove {name} from uploads: {e}"),
+        }
+    }
+
+    Ok(swept)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{run, Swept};
+
+    #[tokio::test]
+    async fn only_what_nothing_knows_about_goes() {
+        let db = sqlx::sqlite::SqlitePoolOptions::new()
+            .max_connections(1)
+            .connect("sqlite::memory:")
+            .await
+            .unwrap();
+        sqlx::migrate!("./migrations").run(&db).await.unwrap();
+        let dir = tempfile::tempdir().unwrap();
+
+        let old = crate::now() - 30 * 86_400;
+        let kept = "11111111-1111-4111-8111-111111111111";
+        let embedded = "22222222-2222-4222-8222-222222222222";
+        let forgotten = "33333333-3333-4333-8333-333333333333";
+        let stray = "44444444-4444-4444-8444-444444444444";
+        for id in [kept, embedded, forgotten] {
+            sqlx::query("INSERT INTO media (id, filename, mime, size, created_at) VALUES (?1, 'x', 'image/png', 1, ?2)")
+                .bind(id)
+                .bind(old)
+                .execute(&db)
+                .await
+                .unwrap();
+        }
+        let space: i64 = sqlx::query_scalar("SELECT id FROM nodes WHERE role = 'diary'")
+            .fetch_one(&db)
+            .await
+            .unwrap();
+        sqlx::query(
+            "INSERT INTO nodes (parent_id, kind, name, slug, body, created_at, updated_at)
+             VALUES (?1, 'document', 'a', 'a', ?2, 0, 0)",
+        )
+        .bind(space)
+        .bind(format!("![](/api/media/{embedded})"))
+        .execute(&db)
+        .await
+        .unwrap();
+        // `kept` is linked the way a saved document links what it embeds.
+        sqlx::query("INSERT INTO node_media (node_id, media_id) SELECT id, ?1 FROM nodes WHERE slug = 'a'")
+            .bind(kept)
+            .execute(&db)
+            .await
+            .unwrap();
+
+        let hour_ago = std::time::SystemTime::now() - std::time::Duration::from_secs(2 * 60 * 60);
+        for name in [kept, embedded, forgotten, stray, "x.part", "notes.txt"] {
+            let path = dir.path().join(name);
+            std::fs::write(&path, b"x").unwrap();
+            std::fs::File::options()
+                .write(true)
+                .open(&path)
+                .unwrap()
+                .set_modified(hour_ago)
+                .unwrap();
+        }
+        let fresh = "55555555-5555-4555-8555-555555555555.part";
+        std::fs::write(dir.path().join(fresh), b"x").unwrap();
+
+        let swept = run(&db, dir.path()).await.unwrap();
+        assert_eq!(swept, Swept { partial: 1, stray: 2, unreferenced: 1 });
+
+        let left: std::collections::BTreeSet<String> = std::fs::read_dir(dir.path())
+            .unwrap()
+            .map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
+            .collect();
+        let expected: std::collections::BTreeSet<String> =
+            [kept, embedded, "notes.txt", fresh].iter().map(|s| s.to_string()).collect();
+        assert_eq!(left, expected);
+    }
+}

web/src/lib/MediaBrowser.svelte

@@ -5,8 +5,11 @@
   import { workspace } from './store.svelte'
   import { formatBytes, formatStamp } from './util'
 
-  const usage = (nodeIds: number[]) =>
+  const usage = (nodeIds: number[], fileNode: number | null) => {
-    nodeIds.length === 0 ? 'unattached' : nodeIds.map((id) => `#${id}`).join(' ')
+    const embeds = nodeIds.map((id) => `#${id}`).join(' ')
+    if (fileNode !== null) return embeds ? `in the tree · ${embeds}` : 'in the tree'
+    return embeds || 'unattached'
+  }
 
   async function remove(id: string, filename: string, nodeIds: number[]) {
     const used = nodeIds.length === 1 ? '1 document' : `${nodeIds.length} documents`
@@ -42,7 +45,7 @@
           <span class="name">{file.filename}</span>
           <span class="muted">
             {formatBytes(file.size)} · {formatStamp(file.created_at)} ·
-            {usage(file.node_ids)}
+            {usage(file.node_ids, file.file_node)}
           </span>
         </div>
 
@@ -56,7 +59,9 @@
             >
           {/if}
           <button onclick={() => workspace.copy(location.origin + file.url)}>copy url</button>
+          {#if file.file_node === null}
             <button class="danger" onclick={() => remove(file.id, file.filename, file.node_ids)}>rm</button>
+          {/if}
         </div>
       </li>
     {:else}

web/src/lib/api.ts

@@ -23,6 +23,8 @@ export interface NodeSummary {
   excerpt: string
   position: number
   has_board: boolean
+  /** Spaces the app has to tell apart: `diary`, or nothing. */
+  role: string | null
   child_count: number
   created_at: number
   updated_at: number
@@ -45,13 +47,13 @@ export interface WorkspaceNode extends NodeSummary {
   path: Crumb[]
 }
 
-/** A public key allowed to mount the workspace over SFTP. */
 /** A space as the overview shows it: itself, and what was last written in it. */
 export interface SpaceOverview {
   space: NodeSummary
   recent: NodeSummary[]
 }
 
+/** A public key allowed to mount the workspace over SFTP. */
 export interface SshKey {
   id: number
   name: string
@@ -130,6 +132,8 @@ export interface MediaFile {
   encrypted: boolean
   /** Every document that embeds this file — a file may be used by several. */
   node_ids: number[]
+  /** The file node it is, when it sits in the tree; deleted from there. */
+  file_node: number | null
 }
 
 /** The vault, as `/api/vault` reports it. */

web/src/lib/commands.ts

@@ -382,7 +382,7 @@ export const commands: CommandSpec[] = [
       await workspace.guard(async () => {
         // The diary is one space among several now, so :today goes there first
         // rather than writing the day into whatever folder was being browsed.
-        const home = workspace.spaces.find((space) => space.slug === 'diary')
+        const home = workspace.spaces.find((space) => space.role === 'diary')
         if (home && workspace.parentId !== home.id) await workspace.enterSpace(home.id)
         else if (workspace.query) await workspace.search('')
 

web/src/lib/store.svelte.ts

@@ -448,7 +448,7 @@ class Workspace {
     const remembered = Number(localStorage.getItem('diary:space'))
     const start =
       this.spaces.find((s) => s.id === remembered) ??
-      this.spaces.find((s) => s.slug === 'diary') ??
+      this.spaces.find((s) => s.role === 'diary') ??
       this.spaces[0]
     if (start) await this.enterSpace(start.id)
   }
@@ -722,7 +722,7 @@ class Workspace {
     this.rawBoard = null
     this.view = 'tree'
     this.spaces = await api.listSpaces()
-    const next = this.spaces.find((space) => space.slug === 'diary') ?? this.spaces[0]
+    const next = this.spaces.find((space) => space.role === 'diary') ?? this.spaces[0]
     if (next) await this.enterSpace(next.id)
     else {
       localStorage.removeItem('diary:space')