mirror of
https://github.com/narrrl/narl-workspace.git
synced 2026-11-03 16:41:27 +01:00
export, import: an export that pours back into a workspace whole
An export used to lose what plain files cannot carry: dates, names as typed, the boards. It now writes .workspace.json beside the tree (format 1), and an import that finds one restores them. The markdown stays plain, so the archive still reads without the application. POST /api/import (:import! / in the UI) imports at the root: each top-level directory becomes a space, or merges into the space already there by that name, with its board and cards and the diary role. With a manifest, media/ comes back as embedded files rather than a folder, a file whose name ends in .md stays a file, and empty folders come back. Export claims every path in one set, folders, then documents, then files, so a file named notes.md beside a document notes no longer makes a duplicate entry that failed the whole zip. Links to folders are rewritten like links to documents, in one regex pass per kind instead of a replace per file and document. A sealed file that will not open now fails the export instead of being left out silently. The temporary zip is made in the data directory, not in /tmp.
This commit is contained in:
1 parent
b1ba63dc61
commit
aa34be8663
10 files changed
+799
-199
No files matched your search
@@ -158,10 +158,13 @@ it is removed again on the way out, so abandoned starts do not pile up. If a tab
|
||||
says so — `:e!` throws it away, the way vim handles a swap file.
|
||||
|
||||
`:export` downloads the open document as markdown. `:export!` downloads the
|
||||
whole workspace as a zip: the tree as real directories, one markdown file per
|
||||
document, every file in the tree where it sits and every other embedded file in
|
||||
`media/`, under their real names, and relative links between them, so the
|
||||
archive reads in any markdown viewer without this application.
|
||||
whole workspace as a zip: the tree as real directories, empty folders too, one
|
||||
markdown file per document, every file in the tree where it sits and every
|
||||
other embedded file in `media/`, under their real names, and relative links
|
||||
between documents, folders and files, so the archive reads in any markdown
|
||||
viewer without this application. What plain files have no place for — when
|
||||
each entry was written, names as they were typed, which space is the diary,
|
||||
every board with its cards — goes into one `.workspace.json` beside them.
|
||||
|
||||
The diary is the diary by role, not by name: rename it and the reminder and
|
||||
the agenda still know which space it is, and it cannot be deleted — empty it
|
||||
@@ -237,8 +240,12 @@ Proton cannot read either; only the vault password can read the sealed one.
|
||||
had inside, and the tree is rebuilt on the other side: directories become
|
||||
folders, `.md` and `.markdown` files become documents (titled by their first
|
||||
heading), anything else becomes a file in the tree. `:import!` does the same with
|
||||
a zip, which is the shape `:export!` writes, so a workspace can be poured back
|
||||
into another one.
|
||||
a zip, which is the shape `:export!` writes. `:import! /` pours an `:export!`
|
||||
back in at the root: each directory becomes a space again — or goes into the
|
||||
space already there by that name — and with `.workspace.json` beside it the
|
||||
dates, names, boards and cards come back as they were, embedded files go back
|
||||
to being embedded rather than a `media` folder, and a file that happened to end
|
||||
in `.md` stays a file.
|
||||
|
||||
Links between the imported files are rewritten as they land: a link to a
|
||||
markdown file becomes the document's address, a link to an image becomes its
|
||||
@@ -433,9 +440,10 @@ Everything except the two share routes requires the session cookie.
|
||||
| `POST` | `/api/cards/{id}/move` | to another list and place in it |
|
||||
| `POST` | `/api/nodes/{id}/move` | reparent, and place among the new siblings |
|
||||
| `POST` | `/api/nodes/{id}/import` | import a folder or zip (multipart) under this node |
|
||||
| `POST` | `/api/import` | import an export at the root: directories become spaces |
|
||||
| `GET` | `/api/resolve?path=` | the node behind a slug path, `work/notes/monday` |
|
||||
| `POST`/`DELETE` | `/api/nodes/{id}/share` | mint or rotate / revoke a share link |
|
||||
| `GET` | `/api/export` | every document and file, as a zip |
|
||||
| `GET` | `/api/export` | the whole workspace, boards and dates included, as a zip |
|
||||
| `GET`/`POST` | `/api/backup` | Proton Drive mirror: status / run now |
|
||||
| `GET`/`POST` | `/api/media` | list / upload (multipart) |
|
||||
| `GET`/`DELETE` | `/api/media/{id}` | serve / delete a file |
|
||||
|
||||
+91
-1
@@ -20,13 +20,18 @@ use axum::{
|
||||
extract::{Path, State},
|
||||
Json,
|
||||
};
|
||||
use std::collections::HashMap;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sqlx::{Row, SqliteConnection, SqlitePool};
|
||||
|
||||
use crate::{
|
||||
auth::Session,
|
||||
error::{AppError, AppResult},
|
||||
routes::nodes::{DOCUMENT, SPACE},
|
||||
routes::{
|
||||
manifest::ListMeta,
|
||||
nodes::{DOCUMENT, SPACE},
|
||||
},
|
||||
state::AppState,
|
||||
};
|
||||
|
||||
@@ -351,6 +356,91 @@ async fn settle(db: &mut SqliteConnection, space_id: i64) -> AppResult<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Put the lists and cards of an exported board back on a space. An empty
|
||||
/// board takes the lists as they were; one that has lists already keeps them,
|
||||
/// gains the ones it lacks in front of its last — the way `:list` adds one —
|
||||
/// and takes each card into the list of the same name. Returns the cards.
|
||||
pub(crate) async fn restore(
|
||||
db: &mut SqliteConnection,
|
||||
space_id: i64,
|
||||
lists: &[ListMeta],
|
||||
documents: &HashMap<String, i64>,
|
||||
) -> AppResult<usize> {
|
||||
if lists.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let existing: Vec<(i64, String)> =
|
||||
sqlx::query("SELECT id, name FROM board_lists WHERE space_id = ?1 ORDER BY position, id")
|
||||
.bind(space_id)
|
||||
.fetch_all(&mut *db)
|
||||
.await?
|
||||
.iter()
|
||||
.map(|row| (row.get("id"), row.get("name")))
|
||||
.collect();
|
||||
|
||||
let now = crate::now();
|
||||
let mut ids = Vec::with_capacity(lists.len());
|
||||
for (index, list) in lists.iter().enumerate() {
|
||||
if let Some((id, _)) = existing.iter().find(|(_, name)| *name == list.name) {
|
||||
ids.push(*id);
|
||||
continue;
|
||||
}
|
||||
let position = match last_list(&mut *db, space_id).await? {
|
||||
// In front of "done", which has to stay last.
|
||||
Some((last, position)) if !existing.is_empty() => {
|
||||
sqlx::query("UPDATE board_lists SET position = position + 1 WHERE id = ?1")
|
||||
.bind(last)
|
||||
.execute(&mut *db)
|
||||
.await?;
|
||||
position
|
||||
}
|
||||
_ => index as i64,
|
||||
};
|
||||
let id: i64 = sqlx::query_scalar(
|
||||
"INSERT INTO board_lists (space_id, name, position, created_at)
|
||||
VALUES (?1, ?2, ?3, ?4) RETURNING id",
|
||||
)
|
||||
.bind(space_id)
|
||||
.bind(&list.name)
|
||||
.bind(position)
|
||||
.bind(now)
|
||||
.fetch_one(&mut *db)
|
||||
.await?;
|
||||
ids.push(id);
|
||||
}
|
||||
|
||||
let mut count = 0;
|
||||
for (list, id) in lists.iter().zip(ids) {
|
||||
let mut position: i64 =
|
||||
sqlx::query_scalar("SELECT COALESCE(MAX(position), -1) + 1 FROM cards WHERE list_id = ?1")
|
||||
.bind(id)
|
||||
.fetch_one(&mut *db)
|
||||
.await?;
|
||||
for card in &list.cards {
|
||||
sqlx::query(
|
||||
"INSERT INTO cards (list_id, node_id, title, body, due_at, done_at, position,
|
||||
created_at, updated_at)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(card.document.as_ref().and_then(|path| documents.get(path)))
|
||||
.bind(&card.title)
|
||||
.bind(&card.body)
|
||||
.bind(card.due_at)
|
||||
.bind(card.done_at)
|
||||
.bind(position)
|
||||
.bind(card.created_at)
|
||||
.bind(card.updated_at)
|
||||
.execute(&mut *db)
|
||||
.await?;
|
||||
position += 1;
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
settle(db, space_id).await?;
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
/// A card links to a document — its notes — and to nothing else: a folder
|
||||
/// cannot be opened to write in, and a file has no text to hold them.
|
||||
async fn linkable(db: &SqlitePool, node_id: Option<i64>) -> AppResult<()> {
|
||||
|
||||
+199
-55
@@ -1,6 +1,7 @@
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
collections::{HashMap, HashSet},
|
||||
io::{Seek, SeekFrom, Write},
|
||||
sync::LazyLock,
|
||||
};
|
||||
|
||||
use axum::{
|
||||
@@ -9,6 +10,7 @@ use axum::{
|
||||
http::header,
|
||||
response::{IntoResponse, Response},
|
||||
};
|
||||
use regex::Regex;
|
||||
use sqlx::Row;
|
||||
use tokio_util::io::ReaderStream;
|
||||
use zip::{write::SimpleFileOptions, CompressionMethod, ZipWriter};
|
||||
@@ -16,9 +18,20 @@ use zip::{write::SimpleFileOptions, CompressionMethod, ZipWriter};
|
||||
use crate::{
|
||||
auth::Session,
|
||||
error::{AppError, AppResult},
|
||||
routes::{
|
||||
manifest::{self, CardMeta, ListMeta, Manifest, NodeMeta, SpaceMeta},
|
||||
media::MEDIA_REF,
|
||||
nodes::{is_container, DOCUMENT, SPACE},
|
||||
},
|
||||
state::AppState,
|
||||
};
|
||||
|
||||
/// A markdown link to an address inside the application, `](/n/work/notes)`,
|
||||
/// up to the `)` or `#` that ends it — which is what keeps `/n/work/a` from
|
||||
/// matching inside `/n/work/ab`.
|
||||
static NODE_LINK: LazyLock<Regex> =
|
||||
LazyLock::new(|| Regex::new(r"\]\((/n/[^)#\s]+)([)#])").expect("static regex is valid"));
|
||||
|
||||
/// One node, flattened out of the database ahead of the blocking zip work.
|
||||
struct ExportNode {
|
||||
id: i64,
|
||||
@@ -28,6 +41,15 @@ struct ExportNode {
|
||||
slug: String,
|
||||
body: String,
|
||||
media_id: Option<String>,
|
||||
created_at: i64,
|
||||
updated_at: i64,
|
||||
has_board: bool,
|
||||
role: Option<String>,
|
||||
}
|
||||
|
||||
/// A container's directory in the archive.
|
||||
fn container_path(nodes: &HashMap<i64, ExportNode>, node: &ExportNode) -> String {
|
||||
tree_path(nodes, node.parent_id, node.slug.clone())
|
||||
}
|
||||
|
||||
/// Where a document lands in the archive: the slugs of its ancestors as
|
||||
@@ -130,10 +152,12 @@ fn file_path(
|
||||
const README: &str = "\
|
||||
This is a complete copy of a ~/workspace.
|
||||
|
||||
<space>/ one directory per space, holding the same tree of folders,
|
||||
documents and files that the workspace shows; one markdown file
|
||||
per document, and every file under its own name
|
||||
media/ the files documents embed that are not in the tree themselves
|
||||
<space>/ one directory per space, holding the same tree of folders,
|
||||
documents and files that the workspace shows; one markdown
|
||||
file per document, and every file under its own name
|
||||
media/ the files documents embed that are not in the tree themselves
|
||||
.workspace.json what plain files cannot hold: dates, names as they were
|
||||
typed, the boards. Importing the archive puts it all back.
|
||||
|
||||
Embedded media, and links from one document to another, are relative — so the
|
||||
documents render and cross-link correctly in any markdown reader as long as the
|
||||
@@ -144,7 +168,9 @@ tree stays intact. Nothing here needs the workspace application to read it.
|
||||
/// can trust, so this is deliberately a plain archive of plain files.
|
||||
pub async fn export(_: Session, State(state): State<AppState>) -> AppResult<Response> {
|
||||
let nodes: HashMap<i64, ExportNode> = sqlx::query(
|
||||
"SELECT id, parent_id, kind, name, slug, body, media_id FROM nodes ORDER BY created_at, id",
|
||||
"SELECT id, parent_id, kind, name, slug, body, media_id, created_at, updated_at,
|
||||
has_board, role
|
||||
FROM nodes ORDER BY created_at, id",
|
||||
)
|
||||
.fetch_all(&state.db)
|
||||
.await?
|
||||
@@ -158,6 +184,10 @@ pub async fn export(_: Session, State(state): State<AppState>) -> AppResult<Resp
|
||||
slug: row.get("slug"),
|
||||
body: row.get("body"),
|
||||
media_id: row.get("media_id"),
|
||||
created_at: row.get("created_at"),
|
||||
updated_at: row.get("updated_at"),
|
||||
has_board: row.get("has_board"),
|
||||
role: row.get("role"),
|
||||
};
|
||||
(node.id, node)
|
||||
})
|
||||
@@ -170,12 +200,40 @@ pub async fn export(_: Session, State(state): State<AppState>) -> AppResult<Resp
|
||||
.iter()
|
||||
.map(|row| (row.get("id"), row.get("filename")))
|
||||
.collect();
|
||||
let sealed: std::collections::HashSet<String> = rows
|
||||
let sealed: HashSet<String> = rows
|
||||
.iter()
|
||||
.filter(|row| row.get::<bool, _>("encrypted"))
|
||||
.map(|row| row.get("id"))
|
||||
.collect();
|
||||
|
||||
let lists: Vec<(i64, i64, String)> =
|
||||
sqlx::query("SELECT id, space_id, name FROM board_lists ORDER BY position, id")
|
||||
.fetch_all(&state.db)
|
||||
.await?
|
||||
.iter()
|
||||
.map(|row| (row.get("id"), row.get("space_id"), row.get("name")))
|
||||
.collect();
|
||||
let cards: Vec<(i64, CardMeta, Option<i64>)> = sqlx::query(
|
||||
"SELECT list_id, title, body, due_at, done_at, created_at, updated_at, node_id
|
||||
FROM cards ORDER BY position, id",
|
||||
)
|
||||
.fetch_all(&state.db)
|
||||
.await?
|
||||
.iter()
|
||||
.map(|row| {
|
||||
let card = CardMeta {
|
||||
title: row.get("title"),
|
||||
body: row.get("body"),
|
||||
due_at: row.get("due_at"),
|
||||
done_at: row.get("done_at"),
|
||||
created_at: row.get("created_at"),
|
||||
updated_at: row.get("updated_at"),
|
||||
document: None,
|
||||
};
|
||||
(row.get("list_id"), card, row.get("node_id"))
|
||||
})
|
||||
.collect();
|
||||
|
||||
// The archive is meant to be read without this application, so sealed
|
||||
// files go in opened — which needs the vault open. Leaving them out
|
||||
// instead would make an export that silently is not complete.
|
||||
@@ -186,12 +244,15 @@ pub async fn export(_: Session, State(state): State<AppState>) -> AppResult<Resp
|
||||
};
|
||||
|
||||
let uploads = state.config.uploads_dir();
|
||||
let scratch = state.config.data_dir.clone();
|
||||
|
||||
// Zip writing is synchronous and reads every upload off disk, so it happens
|
||||
// on the blocking pool, into a temporary file rather than into memory: a
|
||||
// diary with photographs in it does not fit comfortably in a Vec.
|
||||
let file = tokio::task::spawn_blocking(move || -> anyhow::Result<std::fs::File> {
|
||||
let mut zip = ZipWriter::new(tempfile::tempfile()?);
|
||||
// diary with photographs in it does not fit comfortably in a Vec. The file
|
||||
// is made in the data directory, which is sized for the workspace, rather
|
||||
// than in a /tmp that may be a small tmpfs.
|
||||
let file = tokio::task::spawn_blocking(move || -> AppResult<std::fs::File> {
|
||||
let mut zip = ZipWriter::new(tempfile::tempfile_in(&scratch)?);
|
||||
let text = SimpleFileOptions::default().compression_method(CompressionMethod::Deflated);
|
||||
// Media is overwhelmingly already-compressed; deflating it again costs
|
||||
// a lot of CPU to save nothing.
|
||||
@@ -200,88 +261,167 @@ pub async fn export(_: Session, State(state): State<AppState>) -> AppResult<Resp
|
||||
zip.start_file("README.txt", text)?;
|
||||
zip.write_all(README.as_bytes())?;
|
||||
|
||||
// A file node's bytes go where the node is; the rest go to media/.
|
||||
// Ordered by id, so a clash of names resolves the same way every time.
|
||||
let mut files: Vec<&ExportNode> = nodes.values().filter(|n| n.media_id.is_some()).collect();
|
||||
files.sort_by_key(|n| n.id);
|
||||
let mut taken = std::collections::HashSet::new();
|
||||
let in_tree: HashMap<String, String> = files
|
||||
.into_iter()
|
||||
.filter_map(|n| Some((n.media_id.clone()?, file_path(&nodes, n, &mut taken))))
|
||||
// Every path the archive will hold, claimed in one set: directories
|
||||
// first, then documents, then files — the only ones that can give
|
||||
// way, by taking their id — so no two entries ever share a name.
|
||||
let mut ordered: Vec<&ExportNode> = nodes.values().collect();
|
||||
ordered.sort_by_key(|n| n.id);
|
||||
let mut taken: HashSet<String> = HashSet::from(["README.txt".into(), manifest::FILE_NAME.into()]);
|
||||
let containers: Vec<(&ExportNode, String)> = ordered
|
||||
.iter()
|
||||
.filter(|n| is_container(&n.kind))
|
||||
.map(|n| (*n, container_path(&nodes, n)))
|
||||
.collect();
|
||||
taken.extend(containers.iter().map(|(_, path)| path.clone()));
|
||||
let documents: Vec<(&ExportNode, String)> = ordered
|
||||
.iter()
|
||||
.filter(|n| n.kind == DOCUMENT)
|
||||
.map(|n| (*n, document_path(&nodes, n)))
|
||||
.collect();
|
||||
taken.extend(documents.iter().map(|(_, path)| path.clone()));
|
||||
let in_tree: HashMap<String, (&ExportNode, String)> = ordered
|
||||
.iter()
|
||||
.filter_map(|n| Some((n.media_id.clone()?, (*n, file_path(&nodes, n, &mut taken)))))
|
||||
.collect();
|
||||
|
||||
let names: Vec<(String, String)> = media
|
||||
let names: HashMap<String, String> = media
|
||||
.iter()
|
||||
.filter(|(id, _)| !in_tree.contains_key(id))
|
||||
.map(|(id, filename)| (id.clone(), media_filename(id, filename)))
|
||||
.collect();
|
||||
|
||||
// Every document by the address the application knows it as, so a link
|
||||
// between two of them can be found and pointed at the file instead.
|
||||
let addresses: HashMap<String, String> = nodes
|
||||
.values()
|
||||
.filter(|n| n.kind == "document")
|
||||
.map(|n| {
|
||||
let path = document_path(&nodes, n);
|
||||
(format!("/n/{}", path.trim_end_matches(".md")), path)
|
||||
})
|
||||
// Every document and folder by the address the application knows it
|
||||
// as, so a link to one can be found and pointed at the file instead.
|
||||
let addresses: HashMap<String, String> = documents
|
||||
.iter()
|
||||
.map(|(_, path)| (format!("/n/{}", path.trim_end_matches(".md")), path.clone()))
|
||||
.chain(containers.iter().map(|(_, path)| (format!("/n/{path}"), path.clone())))
|
||||
.collect();
|
||||
|
||||
for node in nodes.values().filter(|n| n.kind == "document") {
|
||||
let path = document_path(&nodes, node);
|
||||
let mut meta = Manifest { format: manifest::FORMAT, ..Manifest::default() };
|
||||
let mut note = |path: &str, node: &ExportNode| {
|
||||
meta.nodes.insert(
|
||||
path.to_string(),
|
||||
NodeMeta {
|
||||
kind: node.kind.clone(),
|
||||
name: node.name.clone(),
|
||||
created_at: node.created_at,
|
||||
updated_at: node.updated_at,
|
||||
},
|
||||
);
|
||||
};
|
||||
|
||||
for (node, path) in &containers {
|
||||
note(path, node);
|
||||
// An empty folder is still part of the tree.
|
||||
zip.add_directory(format!("{path}/"), text)?;
|
||||
}
|
||||
|
||||
for (node, path) in &documents {
|
||||
note(path, node);
|
||||
// A document three directories deep needs three steps back up to
|
||||
// reach the media folder next to the spaces.
|
||||
let up = "../".repeat(path.matches('/').count());
|
||||
|
||||
let mut body = node.body.clone();
|
||||
for (id, name) in &names {
|
||||
body = body.replace(&format!("/api/media/{id}"), &format!("{up}media/{name}"));
|
||||
}
|
||||
for (id, target) in &in_tree {
|
||||
body = body.replace(&format!("/api/media/{id}"), &relative_link(&path, target));
|
||||
}
|
||||
for (address, target) in &addresses {
|
||||
// Matching the `)` or `#` that ends a markdown link is what
|
||||
// keeps `/n/work/a` from matching inside `/n/work/ab`.
|
||||
body = body.replace(&format!("]({address})"), &format!("]({})", relative_link(&path, target)));
|
||||
body = body.replace(&format!("]({address}#"), &format!("]({}#", relative_link(&path, target)));
|
||||
}
|
||||
// One pass for each kind of link, looking every match up, rather
|
||||
// than one pass per file and per document the workspace holds.
|
||||
let body = MEDIA_REF.replace_all(&node.body, |caps: ®ex::Captures| {
|
||||
let id = caps[1].to_lowercase();
|
||||
if let Some(name) = names.get(&id) {
|
||||
format!("{up}{}/{name}", manifest::MEDIA_DIR)
|
||||
} else if let Some((_, target)) = in_tree.get(&id) {
|
||||
relative_link(path, target)
|
||||
} else {
|
||||
caps[0].to_string()
|
||||
}
|
||||
});
|
||||
let body = NODE_LINK.replace_all(&body, |caps: ®ex::Captures| match addresses.get(&caps[1]) {
|
||||
Some(target) => format!("]({}{}", relative_link(path, target), &caps[2]),
|
||||
None => caps[0].to_string(),
|
||||
});
|
||||
let document = if node.name.trim().is_empty() {
|
||||
body
|
||||
body.into_owned()
|
||||
} else {
|
||||
format!("# {}\n\n{body}", node.name.trim())
|
||||
};
|
||||
|
||||
zip.start_file(path, text)?;
|
||||
zip.start_file(path.as_str(), text)?;
|
||||
zip.write_all(document.as_bytes())?;
|
||||
}
|
||||
|
||||
let entries = names
|
||||
for (node, path) in in_tree.values() {
|
||||
note(path, node);
|
||||
}
|
||||
|
||||
let mut entries: Vec<(&String, String)> = names
|
||||
.iter()
|
||||
.map(|(id, name)| (id, format!("media/{name}")))
|
||||
.chain(in_tree.iter().map(|(id, path)| (id, path.clone())));
|
||||
.map(|(id, name)| (id, format!("{}/{name}", manifest::MEDIA_DIR)))
|
||||
.chain(in_tree.iter().map(|(id, (_, path))| (id, path.clone())))
|
||||
.collect();
|
||||
entries.sort();
|
||||
for (id, entry) in entries {
|
||||
// A row whose file has gone missing must not fail the whole backup.
|
||||
let path = uploads.join(id);
|
||||
let source: Box<dyn std::io::Read> = match (&key, sealed.contains(id)) {
|
||||
(Some(key), true) => match crate::vault::Reader::open(key, id, &path) {
|
||||
Ok(reader) => Box::new(reader),
|
||||
Err(_) => continue,
|
||||
},
|
||||
// A sealed file that will not open is not a file to leave out
|
||||
// quietly: the archive would claim to be complete and not be.
|
||||
(Some(key), true) => Box::new(crate::vault::Reader::open(key, id, &path).map_err(|e| {
|
||||
AppError::BadRequest(format!("{entry} is sealed and would not open ({e}) — nothing was exported"))
|
||||
})?),
|
||||
// A row whose file has gone missing must not fail the whole
|
||||
// export; there is nothing to put in its place either way.
|
||||
_ => match std::fs::File::open(&path) {
|
||||
Ok(file) => Box::new(file),
|
||||
Err(_) => continue,
|
||||
Err(e) => {
|
||||
tracing::warn!("export: {entry} has no file on disk ({e}), left out");
|
||||
continue;
|
||||
}
|
||||
},
|
||||
};
|
||||
if let Some(original) = media.iter().find(|(m, _)| m == id).map(|(_, f)| f) {
|
||||
if !in_tree.contains_key(id) {
|
||||
meta.media.insert(entry.clone(), original.clone());
|
||||
}
|
||||
}
|
||||
zip.start_file(entry, blob)?;
|
||||
std::io::copy(&mut { source }, &mut zip)?;
|
||||
}
|
||||
|
||||
// The boards, with each card's notes named by where they sit in here.
|
||||
let paths: HashMap<i64, &String> = documents.iter().map(|(n, path)| (n.id, path)).collect();
|
||||
for (node, path) in &containers {
|
||||
if node.kind != SPACE || node.parent_id.is_some() {
|
||||
continue;
|
||||
}
|
||||
let lists = lists
|
||||
.iter()
|
||||
.filter(|(_, space, _)| *space == node.id)
|
||||
.map(|(id, _, name)| ListMeta {
|
||||
name: name.clone(),
|
||||
cards: cards
|
||||
.iter()
|
||||
.filter(|(list, _, _)| list == id)
|
||||
.map(|(_, card, document)| CardMeta {
|
||||
document: document.and_then(|d| paths.get(&d).map(|p| (*p).clone())),
|
||||
..card.clone()
|
||||
})
|
||||
.collect(),
|
||||
})
|
||||
.collect();
|
||||
meta.spaces.insert(
|
||||
path.clone(),
|
||||
SpaceMeta { has_board: node.has_board, role: node.role.clone(), lists },
|
||||
);
|
||||
}
|
||||
|
||||
zip.start_file(manifest::FILE_NAME, text)?;
|
||||
serde_json::to_writer_pretty(&mut zip, &meta).map_err(anyhow::Error::from)?;
|
||||
|
||||
let mut file = zip.finish()?;
|
||||
file.seek(SeekFrom::Start(0))?;
|
||||
Ok(file)
|
||||
})
|
||||
.await??;
|
||||
.await
|
||||
.map_err(anyhow::Error::from)??;
|
||||
|
||||
let size = file.metadata()?.len();
|
||||
let name = format!("workspace-{}.zip", day_string(crate::now()));
|
||||
@@ -328,6 +468,10 @@ mod tests {
|
||||
slug: (*slug).into(),
|
||||
body: String::new(),
|
||||
media_id: (*kind == "file").then(|| format!("m{id}")),
|
||||
created_at: 0,
|
||||
updated_at: 0,
|
||||
has_board: false,
|
||||
role: None,
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
+384
-128
@@ -16,7 +16,6 @@ use axum::{
|
||||
};
|
||||
use regex::Regex;
|
||||
use serde::Serialize;
|
||||
use sqlx::Row;
|
||||
use std::sync::LazyLock;
|
||||
use uuid::Uuid;
|
||||
|
||||
@@ -25,7 +24,9 @@ use crate::{
|
||||
error::{AppError, AppResult},
|
||||
routes::{
|
||||
media,
|
||||
nodes::{self, next_slug, slugify, DOCUMENT, FILE, FOLDER},
|
||||
boards,
|
||||
manifest::{self, Manifest, NodeMeta},
|
||||
nodes::{self, next_slug, slugify, DOCUMENT, FILE, FOLDER, SPACE},
|
||||
},
|
||||
state::AppState,
|
||||
};
|
||||
@@ -41,10 +42,13 @@ static LINK: LazyLock<Regex> = LazyLock::new(|| {
|
||||
/// status line has one line, and the tree is right there to look at.
|
||||
#[derive(Serialize)]
|
||||
pub struct ImportReport {
|
||||
/// New spaces, made by an import at the root.
|
||||
pub spaces: usize,
|
||||
pub folders: usize,
|
||||
pub documents: usize,
|
||||
pub files: usize,
|
||||
pub links: usize,
|
||||
pub cards: usize,
|
||||
/// Paths that were refused, so a silently missing file is never a surprise.
|
||||
pub skipped: Vec<String>,
|
||||
}
|
||||
@@ -58,6 +62,8 @@ struct Upload {
|
||||
enum Content {
|
||||
/// A markdown file, read into memory: it is about to become a document body.
|
||||
Markdown(String),
|
||||
/// The `.workspace.json` of an export, read into memory to be parsed.
|
||||
Manifest(String),
|
||||
/// Anything else, already streamed to `uploads/<id>`.
|
||||
File { id: String, mime: String, size: u64 },
|
||||
}
|
||||
@@ -101,6 +107,19 @@ fn file_name(path: &str) -> &str {
|
||||
path.rsplit_once('/').map(|(_, name)| name).unwrap_or(path)
|
||||
}
|
||||
|
||||
/// Read into memory rather than streamed to disk: markdown, and the manifest.
|
||||
fn is_text(path: &str) -> bool {
|
||||
is_markdown(path) || file_name(path) == manifest::FILE_NAME
|
||||
}
|
||||
|
||||
fn text_content(path: &str, text: String) -> Content {
|
||||
if file_name(path) == manifest::FILE_NAME {
|
||||
Content::Manifest(text)
|
||||
} else {
|
||||
Content::Markdown(text)
|
||||
}
|
||||
}
|
||||
|
||||
fn is_markdown(path: &str) -> bool {
|
||||
let lower = path.to_ascii_lowercase();
|
||||
lower.ends_with(".md") || lower.ends_with(".markdown")
|
||||
@@ -291,7 +310,7 @@ async fn collect(
|
||||
continue;
|
||||
};
|
||||
|
||||
if is_markdown(&path) {
|
||||
if is_text(&path) {
|
||||
let mut text = Vec::new();
|
||||
while let Some(chunk) = field
|
||||
.chunk()
|
||||
@@ -309,7 +328,7 @@ async fn collect(
|
||||
continue;
|
||||
}
|
||||
let text = String::from_utf8_lossy(&text).to_string();
|
||||
uploads.push(Upload { path, content: Content::Markdown(text) });
|
||||
uploads.push(Upload { content: text_content(&path, text), path });
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -386,7 +405,7 @@ fn unpack(
|
||||
// header that lies about the size cannot run past the limit either.
|
||||
let mut capped = std::io::Read::take(&mut file, limit - unpacked + 1);
|
||||
|
||||
if is_markdown(&path) {
|
||||
if is_text(&path) {
|
||||
let mut text = Vec::new();
|
||||
std::io::copy(&mut capped, &mut text)?;
|
||||
unpacked += text.len() as u64;
|
||||
@@ -398,7 +417,7 @@ fn unpack(
|
||||
continue;
|
||||
}
|
||||
let text = String::from_utf8_lossy(&text).to_string();
|
||||
uploads.push(Upload { path, content: Content::Markdown(text) });
|
||||
uploads.push(Upload { content: text_content(&path, text), path });
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -446,6 +465,13 @@ fn unpacked_too_large(limit: u64) -> AppError {
|
||||
))
|
||||
}
|
||||
|
||||
/// Where an import goes: under a space or folder, or at the root, where the
|
||||
/// top-level directories of the upload are spaces.
|
||||
enum Into {
|
||||
Node(Box<nodes::Node>),
|
||||
Root,
|
||||
}
|
||||
|
||||
/// `POST /api/nodes/{id}/import` — a folder, or a zip of one, becomes a tree
|
||||
/// under the given space or folder.
|
||||
pub async fn import(
|
||||
@@ -460,42 +486,152 @@ pub async fn import(
|
||||
"a folder can only be imported into a space or a folder".into(),
|
||||
));
|
||||
}
|
||||
run(&state, Into::Node(Box::new(target)), multipart).await.map(Json)
|
||||
}
|
||||
|
||||
/// `POST /api/import` — an export poured back in whole: each top-level
|
||||
/// directory becomes a space, or goes into the space that already has its
|
||||
/// name, and the boards, dates and names come back from `.workspace.json`.
|
||||
pub async fn import_root(
|
||||
_: Session,
|
||||
State(state): State<AppState>,
|
||||
multipart: Multipart,
|
||||
) -> AppResult<Json<ImportReport>> {
|
||||
run(&state, Into::Root, multipart).await.map(Json)
|
||||
}
|
||||
|
||||
async fn run(state: &AppState, into: Into, multipart: Multipart) -> AppResult<ImportReport> {
|
||||
// The files land on disk before the transaction so a failed import writes
|
||||
// nothing to the database; the orphans are swept up on the way out.
|
||||
let mut skipped = Vec::new();
|
||||
let mut written: Vec<String> = Vec::new();
|
||||
let result = match collect(&state, multipart, &mut skipped, &mut written).await {
|
||||
Ok(uploads) if uploads.is_empty() => Err(AppError::BadRequest(
|
||||
"nothing importable in that upload".into(),
|
||||
)),
|
||||
Ok(uploads) => build(&state, &target, uploads, skipped).await,
|
||||
Err(e) => Err(e),
|
||||
};
|
||||
let result = async {
|
||||
let mut uploads = collect(state, multipart, &mut skipped, &mut written).await?;
|
||||
let manifest = take_manifest(&mut uploads)?;
|
||||
if let Some(manifest) = &manifest {
|
||||
files_as_they_were(state, manifest, &mut uploads, &mut written).await?;
|
||||
}
|
||||
if uploads.is_empty() {
|
||||
return Err(AppError::BadRequest("nothing importable in that upload".into()));
|
||||
}
|
||||
build(state, &into, uploads, skipped, manifest).await
|
||||
}
|
||||
.await;
|
||||
if result.is_err() {
|
||||
for media_id in &written {
|
||||
let _ = tokio::fs::remove_file(state.config.uploads_dir().join(media_id)).await;
|
||||
media::remove_upload(&state.config.uploads_dir(), media_id).await;
|
||||
}
|
||||
}
|
||||
|
||||
let report = result?;
|
||||
state.backup.signal();
|
||||
Ok(Json(report))
|
||||
Ok(report)
|
||||
}
|
||||
|
||||
/// A file that happens to end in `.md` was a file, not a document, when it
|
||||
/// was exported, and the manifest says so: it goes back to being one.
|
||||
async fn files_as_they_were(
|
||||
state: &AppState,
|
||||
manifest: &Manifest,
|
||||
uploads: &mut [Upload],
|
||||
written: &mut Vec<String>,
|
||||
) -> AppResult<()> {
|
||||
for upload in uploads.iter_mut() {
|
||||
let was_file = manifest.nodes.get(&upload.path).is_some_and(|m| m.kind == FILE);
|
||||
let Content::Markdown(text) = &upload.content else { continue };
|
||||
if !was_file {
|
||||
continue;
|
||||
}
|
||||
let id = Uuid::new_v4().to_string();
|
||||
let path = state.config.uploads_dir().join(&id);
|
||||
let partial = path.with_extension("part");
|
||||
written.push(id.clone());
|
||||
tokio::fs::write(&partial, text.as_bytes()).await?;
|
||||
tokio::fs::File::open(&partial).await?.sync_all().await?;
|
||||
tokio::fs::rename(&partial, &path).await?;
|
||||
upload.content = Content::File {
|
||||
mime: media::mime_of(&upload.path, None),
|
||||
size: text.len() as u64,
|
||||
id,
|
||||
};
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `a/b` from a prefix and a segment, where an empty prefix is the root.
|
||||
fn join(prefix: &str, segment: &str) -> String {
|
||||
if prefix.is_empty() {
|
||||
segment.to_string()
|
||||
} else {
|
||||
format!("{prefix}/{segment}")
|
||||
}
|
||||
}
|
||||
|
||||
/// Take the manifest out of the upload, if there is one, and make every
|
||||
/// other path relative to where it was found: an export uploaded as a
|
||||
/// folder arrives inside a directory of its own.
|
||||
fn take_manifest(uploads: &mut Vec<Upload>) -> AppResult<Option<Manifest>> {
|
||||
let Some(at) = uploads
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter(|(_, u)| matches!(u.content, Content::Manifest(_)))
|
||||
.min_by_key(|(_, u)| u.path.matches('/').count())
|
||||
.map(|(i, _)| i)
|
||||
else {
|
||||
return Ok(None);
|
||||
};
|
||||
let found = uploads.remove(at);
|
||||
let Content::Manifest(text) = found.content else {
|
||||
unreachable!("filtered on the variant above");
|
||||
};
|
||||
let manifest: Manifest = serde_json::from_str(&text).map_err(|e| {
|
||||
AppError::BadRequest(format!("{} is not readable: {e}", manifest::FILE_NAME))
|
||||
})?;
|
||||
if manifest.format > manifest::FORMAT {
|
||||
return Err(AppError::BadRequest(format!(
|
||||
"{} is from a newer version (format {}); update this one first",
|
||||
manifest::FILE_NAME,
|
||||
manifest.format
|
||||
)));
|
||||
}
|
||||
|
||||
let base = parent_of(&found.path);
|
||||
if !base.is_empty() {
|
||||
let prefix = format!("{base}/");
|
||||
for upload in uploads.iter_mut() {
|
||||
if let Some(rest) = upload.path.strip_prefix(&prefix) {
|
||||
upload.path = rest.to_string();
|
||||
}
|
||||
}
|
||||
}
|
||||
// Any other manifest further down is not one to read; it goes in as text.
|
||||
for upload in uploads.iter_mut() {
|
||||
if let Content::Manifest(text) = &mut upload.content {
|
||||
upload.content = Content::Markdown(std::mem::take(text));
|
||||
}
|
||||
}
|
||||
Ok(Some(manifest))
|
||||
}
|
||||
|
||||
async fn build(
|
||||
state: &AppState,
|
||||
target: &nodes::Node,
|
||||
into: &Into,
|
||||
mut uploads: Vec<Upload>,
|
||||
skipped: Vec<String>,
|
||||
mut skipped: Vec<String>,
|
||||
manifest: Option<Manifest>,
|
||||
) -> AppResult<ImportReport> {
|
||||
let uploads_dir = state.config.uploads_dir();
|
||||
|
||||
// The path of the target in slugs, which every rewritten link starts with.
|
||||
let prefix = target
|
||||
.path
|
||||
.iter()
|
||||
.map(|crumb| crumb.slug.as_str())
|
||||
.collect::<Vec<_>>()
|
||||
.join("/");
|
||||
let prefix = match into {
|
||||
Into::Node(target) => target
|
||||
.path
|
||||
.iter()
|
||||
.map(|crumb| crumb.slug.as_str())
|
||||
.collect::<Vec<_>>()
|
||||
.join("/"),
|
||||
Into::Root => String::new(),
|
||||
};
|
||||
|
||||
// Sorted so a parent directory is always created before its children, and
|
||||
// so two runs of the same folder produce the same slugs.
|
||||
@@ -504,64 +640,131 @@ async fn build(
|
||||
let mut seen: BTreeSet<String> = BTreeSet::new();
|
||||
let mut unique = Vec::with_capacity(uploads.len());
|
||||
for upload in uploads {
|
||||
if seen.insert(upload.path.clone()) {
|
||||
// An export's README describes the archive, not the workspace.
|
||||
let boilerplate = manifest.is_some() && upload.path == "README.txt";
|
||||
// At the root only directories can land: everything is in a space.
|
||||
let loose = matches!(into, Into::Root) && !upload.path.contains('/');
|
||||
if seen.insert(upload.path.clone()) && !boilerplate && !loose {
|
||||
unique.push(upload);
|
||||
} else if let Content::File { id, .. } = &upload.content {
|
||||
let _ = tokio::fs::remove_file(state.config.uploads_dir().join(id)).await;
|
||||
continue;
|
||||
}
|
||||
if loose && !boilerplate {
|
||||
skipped.push(upload.path.clone());
|
||||
}
|
||||
if let Content::File { id, .. } = &upload.content {
|
||||
media::remove_upload(&uploads_dir, id).await;
|
||||
}
|
||||
}
|
||||
let uploads = unique;
|
||||
|
||||
// An export keeps the files documents embed in `media/`, beside the spaces;
|
||||
// they come back as what they were, embedded files, not as a folder.
|
||||
let embedded = |path: &str| manifest.is_some() && parent_of(path) == manifest::MEDIA_DIR;
|
||||
let meta = |path: &str| -> Option<&NodeMeta> { manifest.as_ref()?.nodes.get(path) };
|
||||
|
||||
let mut directories: BTreeSet<String> = BTreeSet::new();
|
||||
for upload in &uploads {
|
||||
for upload in uploads.iter().filter(|u| !embedded(&u.path)) {
|
||||
let mut dir = parent_of(&upload.path);
|
||||
while !dir.is_empty() {
|
||||
directories.insert(dir.to_string());
|
||||
dir = parent_of(dir);
|
||||
}
|
||||
}
|
||||
// Empty folders exist in an export only as names in the manifest.
|
||||
if let Some(manifest) = &manifest {
|
||||
for (path, node) in &manifest.nodes {
|
||||
let is_dir = nodes::is_container(&node.kind);
|
||||
if is_dir && normalize_path(path).as_deref() == Some(path.as_str()) {
|
||||
let mut dir = path.as_str();
|
||||
while !dir.is_empty() {
|
||||
directories.insert(dir.to_string());
|
||||
dir = parent_of(dir);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let now = crate::now();
|
||||
let mut tx = state.db.begin().await?;
|
||||
let mut tx = crate::db::begin_write(&state.db).await?;
|
||||
|
||||
// Slugs already taken under each parent, extended as the tree is built —
|
||||
// the nodes being created are not visible to a query until the commit.
|
||||
let mut siblings: HashMap<i64, BTreeSet<String>> = HashMap::new();
|
||||
let existing = sqlx::query("SELECT slug FROM nodes WHERE parent_id IS ?1")
|
||||
.bind(target.id)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
siblings.insert(
|
||||
target.id,
|
||||
existing.iter().map(|row| row.get("slug")).collect(),
|
||||
);
|
||||
|
||||
let mut dir_ids: HashMap<String, i64> = HashMap::from([(String::new(), target.id)]);
|
||||
let mut dir_paths: HashMap<String, String> = HashMap::from([(String::new(), prefix)]);
|
||||
let mut dir_ids: HashMap<String, i64> = HashMap::new();
|
||||
let mut dir_paths: HashMap<String, String> = HashMap::new();
|
||||
if let Into::Node(target) = into {
|
||||
siblings.insert(target.id, nodes::sibling_slugs(&mut tx, Some(target.id), None).await?);
|
||||
dir_ids.insert(String::new(), target.id);
|
||||
dir_paths.insert(String::new(), prefix);
|
||||
}
|
||||
let mut landed: HashMap<String, Landed> = HashMap::new();
|
||||
let mut spaces = 0usize;
|
||||
let mut folders = 0usize;
|
||||
|
||||
for dir in &directories {
|
||||
let parent = dir_ids[parent_of(dir)];
|
||||
let name = file_name(dir);
|
||||
let taken = siblings.entry(parent).or_default();
|
||||
let slug = next_slug(taken, &slugify(name))?;
|
||||
taken.insert(slug.clone());
|
||||
let dated = meta(dir);
|
||||
let name = dated.map_or(file_name(dir), |m| m.name.as_str());
|
||||
let (created, updated) = dated.map_or((now, now), |m| (m.created_at, m.updated_at));
|
||||
|
||||
let id: i64 = sqlx::query(
|
||||
"INSERT INTO nodes (parent_id, kind, name, slug, created_at, updated_at)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?5)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(parent)
|
||||
.bind(FOLDER)
|
||||
.bind(name)
|
||||
.bind(&slug)
|
||||
.bind(now)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?
|
||||
.get("id");
|
||||
let (id, slug) = if parent_of(dir).is_empty() && matches!(into, Into::Root) {
|
||||
// A space: the one already here by that name, or a new one.
|
||||
let slug = slugify(file_name(dir));
|
||||
let existing: Option<i64> =
|
||||
sqlx::query_scalar("SELECT id FROM nodes WHERE parent_id IS NULL AND slug = ?1")
|
||||
.bind(&slug)
|
||||
.fetch_optional(&mut *tx)
|
||||
.await?;
|
||||
let id = match existing {
|
||||
Some(id) => id,
|
||||
None => {
|
||||
let space = manifest.as_ref().and_then(|m| m.spaces.get(dir));
|
||||
let position: i64 = sqlx::query_scalar(
|
||||
"SELECT COALESCE(MAX(position), -1) + 1 FROM nodes WHERE parent_id IS NULL",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
spaces += 1;
|
||||
sqlx::query_scalar(
|
||||
"INSERT INTO nodes (parent_id, kind, name, slug, position, has_board, created_at, updated_at)
|
||||
VALUES (NULL, ?1, ?2, ?3, ?4, ?5, ?6, ?7)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(SPACE)
|
||||
.bind(name)
|
||||
.bind(&slug)
|
||||
.bind(position)
|
||||
.bind(space.is_none_or(|s| s.has_board))
|
||||
.bind(created)
|
||||
.bind(updated)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?
|
||||
}
|
||||
};
|
||||
siblings.insert(id, nodes::sibling_slugs(&mut tx, Some(id), None).await?);
|
||||
(id, slug)
|
||||
} else {
|
||||
let parent = dir_ids[parent_of(dir)];
|
||||
let taken = siblings.entry(parent).or_default();
|
||||
let slug = next_slug(taken, &slugify(file_name(dir)))?;
|
||||
taken.insert(slug.clone());
|
||||
folders += 1;
|
||||
let id = sqlx::query_scalar(
|
||||
"INSERT INTO nodes (parent_id, kind, name, slug, created_at, updated_at)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(parent)
|
||||
.bind(FOLDER)
|
||||
.bind(name)
|
||||
.bind(&slug)
|
||||
.bind(created)
|
||||
.bind(updated)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
(id, slug)
|
||||
};
|
||||
|
||||
let path = format!("{}/{slug}", dir_paths[parent_of(dir)]);
|
||||
let path = join(&dir_paths.get(parent_of(dir)).cloned().unwrap_or_default(), &slug);
|
||||
dir_ids.insert(dir.clone(), id);
|
||||
dir_paths.insert(dir.clone(), path.clone());
|
||||
landed.insert(dir.clone(), Landed::Node(format!("/n/{path}")));
|
||||
@@ -570,14 +773,76 @@ async fn build(
|
||||
// Documents first, with their bodies as uploaded: the rewriting below needs
|
||||
// to know where every document landed before it can point anything at one.
|
||||
let mut documents: Vec<(i64, String, String)> = Vec::new();
|
||||
let mut by_path: HashMap<String, i64> = HashMap::new();
|
||||
let mut files = 0usize;
|
||||
|
||||
for upload in &uploads {
|
||||
let dir = parent_of(&upload.path);
|
||||
let parent = dir_ids[dir];
|
||||
let dated = meta(&upload.path);
|
||||
let (created, updated) = dated.map_or((now, now), |m| (m.created_at, m.updated_at));
|
||||
|
||||
if let Content::Markdown(raw) = &upload.content {
|
||||
let (name, body) = split_title(raw, &upload.path);
|
||||
if let Content::File { id: media_id, mime, size } = &upload.content {
|
||||
let original = manifest
|
||||
.as_ref()
|
||||
.and_then(|m| m.media.get(&upload.path))
|
||||
.map(String::as_str);
|
||||
let filename = original
|
||||
.or(dated.map(|m| m.name.as_str()))
|
||||
.unwrap_or(file_name(&upload.path));
|
||||
sqlx::query(
|
||||
"INSERT INTO media (id, filename, mime, size, created_at)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5)",
|
||||
)
|
||||
.bind(media_id)
|
||||
.bind(filename)
|
||||
.bind(mime)
|
||||
.bind(*size as i64)
|
||||
.bind(created)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
landed.insert(
|
||||
upload.path.clone(),
|
||||
Landed::Media(format!("/api/media/{media_id}")),
|
||||
);
|
||||
if embedded(&upload.path) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// The file sits in the tree under the name it had, and owns its
|
||||
// bytes the way a document owns what it embeds — so deleting the
|
||||
// folder it landed in takes it along.
|
||||
let parent = dir_ids[parent_of(&upload.path)];
|
||||
let taken = siblings.entry(parent).or_default();
|
||||
let slug = next_slug(taken, &slugify(filename))?;
|
||||
taken.insert(slug.clone());
|
||||
|
||||
let node: i64 = sqlx::query_scalar(
|
||||
"INSERT INTO nodes (parent_id, kind, name, slug, created_at, updated_at, media_id)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(parent)
|
||||
.bind(FILE)
|
||||
.bind(filename)
|
||||
.bind(&slug)
|
||||
.bind(created)
|
||||
.bind(updated)
|
||||
.bind(media_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
|
||||
sqlx::query("INSERT INTO node_media (node_id, media_id) VALUES (?1, ?2)")
|
||||
.bind(node)
|
||||
.bind(media_id)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
files += 1;
|
||||
} else if let Content::Markdown(raw) = &upload.content {
|
||||
let dir = parent_of(&upload.path);
|
||||
let parent = dir_ids[dir];
|
||||
let (title, body) = split_title(raw, &upload.path);
|
||||
// The manifest knows the name as it was typed, including none:
|
||||
// a document with no title was exported without a heading.
|
||||
let name = dated.map_or(title, |m| m.name.clone());
|
||||
// The slug follows the file name rather than the title, so a link
|
||||
// someone wrote to `01-was-ist-eebus.md` keeps reading that way.
|
||||
let stem = file_name(&upload.path);
|
||||
@@ -587,9 +852,9 @@ async fn build(
|
||||
let slug = next_slug(taken, &slugify(stem))?;
|
||||
taken.insert(slug.clone());
|
||||
|
||||
let id: i64 = sqlx::query(
|
||||
let id: i64 = sqlx::query_scalar(
|
||||
"INSERT INTO nodes (parent_id, kind, name, slug, body, created_at, updated_at)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?6)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(parent)
|
||||
@@ -597,61 +862,15 @@ async fn build(
|
||||
.bind(&name)
|
||||
.bind(&slug)
|
||||
.bind(&body)
|
||||
.bind(now)
|
||||
.bind(created)
|
||||
.bind(updated)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?
|
||||
.get("id");
|
||||
|
||||
let path = format!("{}/{slug}", dir_paths[dir]);
|
||||
landed.insert(upload.path.clone(), Landed::Node(format!("/n/{path}")));
|
||||
documents.push((id, upload.path.clone(), body));
|
||||
} else if let Content::File { id: media_id, mime, size } = &upload.content {
|
||||
sqlx::query(
|
||||
"INSERT INTO media (id, filename, mime, size, created_at)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5)",
|
||||
)
|
||||
.bind(media_id)
|
||||
.bind(file_name(&upload.path))
|
||||
.bind(mime)
|
||||
.bind(*size as i64)
|
||||
.bind(now)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
// The file sits in the tree under the name it had, and owns its
|
||||
// bytes the way a document owns what it embeds — so deleting the
|
||||
// folder it landed in takes it along.
|
||||
let name = file_name(&upload.path);
|
||||
let taken = siblings.entry(parent).or_default();
|
||||
let slug = next_slug(taken, &slugify(name))?;
|
||||
taken.insert(slug.clone());
|
||||
|
||||
let node: i64 = sqlx::query(
|
||||
"INSERT INTO nodes (parent_id, kind, name, slug, created_at, updated_at, media_id)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?5, ?6)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(parent)
|
||||
.bind(FILE)
|
||||
.bind(name)
|
||||
.bind(&slug)
|
||||
.bind(now)
|
||||
.bind(media_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await?
|
||||
.get("id");
|
||||
|
||||
sqlx::query("INSERT INTO node_media (node_id, media_id) VALUES (?1, ?2)")
|
||||
.bind(node)
|
||||
.bind(media_id)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
landed.insert(
|
||||
upload.path.clone(),
|
||||
Landed::Media(format!("/api/media/{media_id}")),
|
||||
);
|
||||
files += 1;
|
||||
let path = join(&dir_paths[dir], &slug);
|
||||
landed.insert(upload.path.clone(), Landed::Node(format!("/n/{path}")));
|
||||
by_path.insert(upload.path.clone(), id);
|
||||
documents.push((id, upload.path.clone(), body));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -670,27 +889,38 @@ async fn build(
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
media::link_to_node(&mut tx, *id, &rewritten).await?;
|
||||
}
|
||||
|
||||
for capture in media::MEDIA_REF.captures_iter(&rewritten) {
|
||||
let media_id = capture[1].to_lowercase();
|
||||
sqlx::query(
|
||||
"INSERT OR IGNORE INTO node_media (node_id, media_id)
|
||||
SELECT ?1, id FROM media WHERE id = ?2",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(&media_id)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
// The boards come back only where spaces do.
|
||||
let mut cards = 0usize;
|
||||
if let (Into::Root, Some(manifest)) = (into, &manifest) {
|
||||
for (dir, space) in &manifest.spaces {
|
||||
let Some(&space_id) = dir_ids.get(dir) else { continue };
|
||||
cards += boards::restore(&mut tx, space_id, &space.lists, &by_path).await?;
|
||||
if space.role.as_deref() == Some(nodes::DIARY) {
|
||||
// Only where no space has the role yet: there is one diary.
|
||||
sqlx::query(
|
||||
"UPDATE nodes SET role = ?1, has_board = 0
|
||||
WHERE id = ?2 AND NOT EXISTS (SELECT 1 FROM nodes WHERE role = ?1)",
|
||||
)
|
||||
.bind(nodes::DIARY)
|
||||
.bind(space_id)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(ImportReport {
|
||||
folders: directories.len(),
|
||||
spaces,
|
||||
folders,
|
||||
documents: documents.len(),
|
||||
files,
|
||||
links,
|
||||
cards,
|
||||
skipped,
|
||||
})
|
||||
}
|
||||
@@ -701,9 +931,35 @@ mod tests {
|
||||
|
||||
use super::{
|
||||
expand_zip, is_markdown, join_relative, normalize_path, percent_decode, rewrite_links,
|
||||
split_target, split_title, Content, Landed,
|
||||
split_target, split_title, take_manifest, Content, Landed, Upload,
|
||||
};
|
||||
|
||||
fn text(path: &str, content: Content) -> Upload {
|
||||
Upload { path: path.into(), content }
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_export_uploaded_as_a_folder_is_read_from_where_its_manifest_is() {
|
||||
let manifest = r#"{"format": 1, "nodes": {"work/a.md": {"kind": "document", "name": "A", "created_at": 1, "updated_at": 2}}}"#;
|
||||
let mut uploads = vec![
|
||||
text("workspace-2026/.workspace.json", Content::Manifest(manifest.into())),
|
||||
text("workspace-2026/work/a.md", Content::Markdown("a".into())),
|
||||
text("workspace-2026/work/deeper/.workspace.json", Content::Manifest("{}".into())),
|
||||
];
|
||||
let found = take_manifest(&mut uploads).unwrap().unwrap();
|
||||
assert_eq!(found.nodes["work/a.md"].created_at, 1);
|
||||
let paths: Vec<&str> = uploads.iter().map(|u| u.path.as_str()).collect();
|
||||
assert_eq!(paths, ["work/a.md", "work/deeper/.workspace.json"]);
|
||||
// Only the outermost one is read; another is just a file.
|
||||
assert!(matches!(uploads[1].content, Content::Markdown(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_manifest_from_a_newer_version_is_refused() {
|
||||
let mut uploads = vec![text(".workspace.json", Content::Manifest(r#"{"format": 99}"#.into()))];
|
||||
assert!(take_manifest(&mut uploads).is_err());
|
||||
}
|
||||
|
||||
/// A zip with a note and a 4 KiB blob in it, written to `dir/in.zip`.
|
||||
fn archive(dir: &std::path::Path) -> std::path::PathBuf {
|
||||
use std::io::Write;
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
//! `.workspace.json`: what an export knows that its plain files cannot say.
|
||||
//!
|
||||
//! The archive is meant to read without this application, so it holds
|
||||
//! markdown and files and nothing else in them. What a directory of files has
|
||||
//! no place for — when a diary entry was written, a folder's name as typed
|
||||
//! rather than as its slug, which space is the diary, the boards — goes into
|
||||
//! this one file beside them. An import that finds it puts all of that back;
|
||||
//! one that does not still gets the tree.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
pub const FILE_NAME: &str = ".workspace.json";
|
||||
|
||||
/// Bumped when a change would make an older import misread a newer file.
|
||||
/// An import refuses a format it does not know rather than guessing.
|
||||
pub const FORMAT: u32 = 1;
|
||||
|
||||
/// Where embedded files go in the archive, beside the spaces.
|
||||
pub const MEDIA_DIR: &str = "media";
|
||||
|
||||
#[derive(Serialize, Deserialize, Default)]
|
||||
pub struct Manifest {
|
||||
pub format: u32,
|
||||
/// Keyed by archive path: `work/notes` for a folder, `work/notes/a.md` for
|
||||
/// a document, the file's own path for a file, `work` for a space.
|
||||
#[serde(default)]
|
||||
pub nodes: BTreeMap<String, NodeMeta>,
|
||||
/// Keyed by the space's directory.
|
||||
#[serde(default)]
|
||||
pub spaces: BTreeMap<String, SpaceMeta>,
|
||||
/// Embedded files under `media/`, keyed by archive path, with the name
|
||||
/// they were uploaded under.
|
||||
#[serde(default)]
|
||||
pub media: BTreeMap<String, String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone)]
|
||||
pub struct NodeMeta {
|
||||
/// `space`, `folder`, `document` or `file`.
|
||||
pub kind: String,
|
||||
pub name: String,
|
||||
pub created_at: i64,
|
||||
pub updated_at: i64,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone)]
|
||||
pub struct SpaceMeta {
|
||||
pub has_board: bool,
|
||||
#[serde(default)]
|
||||
pub role: Option<String>,
|
||||
/// First to last; the first is the inbox and the last is done.
|
||||
#[serde(default)]
|
||||
pub lists: Vec<ListMeta>,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone)]
|
||||
pub struct ListMeta {
|
||||
pub name: String,
|
||||
#[serde(default)]
|
||||
pub cards: Vec<CardMeta>,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone)]
|
||||
pub struct CardMeta {
|
||||
pub title: String,
|
||||
#[serde(default)]
|
||||
pub body: String,
|
||||
#[serde(default)]
|
||||
pub due_at: Option<i64>,
|
||||
#[serde(default)]
|
||||
pub done_at: Option<i64>,
|
||||
pub created_at: i64,
|
||||
pub updated_at: i64,
|
||||
/// The archive path of the document that holds the card's notes.
|
||||
#[serde(default)]
|
||||
pub document: Option<String>,
|
||||
}
|
||||
@@ -3,6 +3,7 @@ pub mod boards;
|
||||
pub mod nodes;
|
||||
pub mod export;
|
||||
pub mod import;
|
||||
pub mod manifest;
|
||||
pub mod media;
|
||||
pub mod seal;
|
||||
pub mod session;
|
||||
@@ -52,6 +53,7 @@ pub fn api_router(max_upload: usize) -> Router<AppState> {
|
||||
.route("/nodes/{id}", put(nodes::update))
|
||||
.route("/nodes/{id}", delete(nodes::remove))
|
||||
.route("/nodes/{id}/move", post(nodes::move_node))
|
||||
.route("/import", post(import::import_root).layer(uploads))
|
||||
.route("/nodes/{id}/import", post(import::import).layer(uploads))
|
||||
.route("/nodes/{id}/files", post(media::upload_into).layer(uploads))
|
||||
.route("/nodes/{id}/encrypt", post(seal::encrypt))
|
||||
|
||||
+9
-2
@@ -36,6 +36,8 @@
|
||||
let sealUpload = false
|
||||
let folderInput = $state<HTMLInputElement>()
|
||||
let importInput = $state<HTMLInputElement>()
|
||||
// Whether the next archive picked is poured in at the root, `:import! /`.
|
||||
let importRoot = false
|
||||
let shell = $state<HTMLDivElement>()
|
||||
let overlayEl = $state<HTMLDivElement>()
|
||||
let pending = $state('')
|
||||
@@ -53,7 +55,10 @@
|
||||
treeInput?.click()
|
||||
}
|
||||
hooks.pickFolder = () => folderInput?.click()
|
||||
hooks.pickArchive = () => importInput?.click()
|
||||
hooks.pickArchive = (root) => {
|
||||
importRoot = root
|
||||
importInput?.click()
|
||||
}
|
||||
hooks.focusList = () => shell?.focus()
|
||||
hooks.openCommandLine = (initial) => {
|
||||
cmdGlobal = false
|
||||
@@ -156,8 +161,10 @@
|
||||
const input = event.target as HTMLInputElement
|
||||
const files = Array.from(input.files ?? [])
|
||||
input.value = ''
|
||||
const root = importRoot
|
||||
importRoot = false
|
||||
if (files.length === 0) return
|
||||
await workspace.guard(() => workspace.importFolder(files))
|
||||
await workspace.guard(() => workspace.importFolder(files, root))
|
||||
}
|
||||
|
||||
// What the empty pane offers, in the key/value shape of a status readout.
|
||||
|
||||
@@ -72,10 +72,13 @@ export interface SharedDocument {
|
||||
|
||||
/** What an import did, as `POST /api/nodes/{id}/import` reports it. */
|
||||
export interface ImportReport {
|
||||
/** New spaces, made by an import at the root. */
|
||||
spaces: number
|
||||
folders: number
|
||||
documents: number
|
||||
files: number
|
||||
links: number
|
||||
cards: number
|
||||
skipped: string[]
|
||||
}
|
||||
|
||||
@@ -230,6 +233,13 @@ export const api = {
|
||||
return request<ImportReport>(`/nodes/${id}/import`, { method: 'POST', body: form })
|
||||
},
|
||||
|
||||
/** An `:export!` poured back in whole: its directories become spaces. */
|
||||
importRoot: (files: File[]) => {
|
||||
const form = new FormData()
|
||||
for (const file of files) form.append('file', file, file.webkitRelativePath || file.name)
|
||||
return request<ImportReport>('/import', { method: 'POST', body: form })
|
||||
},
|
||||
|
||||
createNode: (input: {
|
||||
parent_id: number
|
||||
kind?: NodeKind
|
||||
|
||||
@@ -10,7 +10,8 @@ export const hooks = {
|
||||
/** Files for the folder being browsed, rather than for the document. */
|
||||
pickTreeFiles: (_encrypt: boolean) => {},
|
||||
pickFolder: () => {},
|
||||
pickArchive: () => {},
|
||||
/** `root`: into the workspace itself, where each directory is a space. */
|
||||
pickArchive: (_root: boolean) => {},
|
||||
focusEditor: () => {},
|
||||
focusList: () => {},
|
||||
insertText: (_text: string) => {},
|
||||
@@ -239,8 +240,8 @@ export const commands: CommandSpec[] = [
|
||||
help: 'upload a folder into this one — markdown becomes documents, links follow',
|
||||
run: () => hooks.pickFolder(),
|
||||
bang: {
|
||||
help: 'import loose files, or a .zip of a folder, instead of a folder',
|
||||
run: () => hooks.pickArchive(),
|
||||
help: 'import loose files, or a .zip of a folder; :import! / pours an :export! back in, spaces and boards and all',
|
||||
run: (arg) => hooks.pickArchive(arg.trim() === '/'),
|
||||
},
|
||||
},
|
||||
{
|
||||
|
||||
@@ -1130,19 +1130,22 @@ class Workspace {
|
||||
* The report is worth a line of its own: an import that quietly skipped half
|
||||
* its files would only be noticed much later.
|
||||
*/
|
||||
async importFolder(files: File[]) {
|
||||
async importFolder(files: File[], root = false) {
|
||||
const parent = this.parentId
|
||||
if (parent === null) return
|
||||
if (parent === null && !root) return
|
||||
this.say(`importing ${files.length} file${files.length === 1 ? '' : 's'} …`)
|
||||
|
||||
const report = await api.importFolder(parent, files)
|
||||
const report = root ? await api.importRoot(files) : await api.importFolder(parent!, files)
|
||||
if (root) this.spaces = await api.listSpaces()
|
||||
await this.refresh()
|
||||
|
||||
const parts = [
|
||||
...(report.spaces > 0 ? [`${report.spaces} space${report.spaces === 1 ? '' : 's'}`] : []),
|
||||
`${report.documents} document${report.documents === 1 ? '' : 's'}`,
|
||||
`${report.folders} folder${report.folders === 1 ? '' : 's'}`,
|
||||
`${report.files} file${report.files === 1 ? '' : 's'}`,
|
||||
`${report.links} link${report.links === 1 ? '' : 's'} rewritten`,
|
||||
...(report.cards > 0 ? [`${report.cards} card${report.cards === 1 ? '' : 's'}`] : []),
|
||||
]
|
||||
if (report.skipped.length > 0) parts.push(`${report.skipped.length} skipped`)
|
||||
this.say(`imported ${parts.join(', ')}`)
|
||||
|
||||
Reference in new issue
Block a user