aboutsummaryrefslogtreecommitdiffhomepage
path: root/org
Commit message (Collapse)AuthorAgeFilesLines
* add .ssp regression test script and reference filesHEADmainRalph Amissah4 days1-0/+155
| | | | | | | | | | | | | | | | | | | test/test-abstraction-ssp.sh: shell script that generates .ssp files for all sample documents and diffs against committed reference files in test/reference/abstraction/. Usage: ./test/test-abstraction-ssp.sh --generate ./bin/spine-ldc ./test/test-abstraction-ssp.sh ./bin/spine-ldc Exits 0 if all .ssp files match, 1 if differences found. Reports changed, missing, and new files. test/reference/abstraction/: 35 reference .ssp files generated from the current abstraction, covering all sample documents including 9-language live-manual. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* add --pod2 flag, decouple --show-abstraction from --podRalph Amissah4 days2-31/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Finer-grained control over when .ssp files are produced: --show-abstraction writes .ssp to OUTPUT/lang/abstraction/ independently of any pod flag --pod builds pod without .ssp bundled --pod2 builds pod with .ssp in media/abstraction/ Changes to spine.d: - show_abstraction() now only responds to its own flag and pod2, no longer triggered by source_or_pod - Add pod2 to opts init, getopt, OptActions - pod() returns true for both --pod and --pod2 - source_or_pod() includes pod2 Changes to source_pod.d: - Remove per-document pod directory (rmdirRecurse) before regeneration, ensuring clean slate on every run. This prevents stale content from previous runs (e.g. a --pod2 run followed by --pod would otherwise leave an outdated media/abstraction/ directory) - Gate abstraction directory creation and .ssp bundling on pod2 flag specifically Tested: --pod (no .ssp), --pod2 (.ssp in pod + zip), --show-abstraction (standalone .ssp), --pod after --pod2 (stale abstraction cleaned up). All 35 sample documents pass. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* .ssp: omit empty-value array property entriesRalph Amissah4 days1-3/+6
| | | | | | | | | | | | | Add empty-string guards to array property loops (.stow_link, .lev4_subtoc, .anchor_tag) so entries with zero-length values are not emitted. Empty properties have no value for PEG parsing - absent lines are faster to skip than matching a property name to find an empty value. Removes 1488 empty .anchor_tag: lines from Wealth of Networks .ssp alone. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* .ssp: add .children property for heading tree navigationRalph Amissah4 days1-0/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | - Add explicit child heading OCN lists to heading objects, pre-computed in a single O(n) pass over the body section before serialization. This makes the document tree directly navigable without scanning - each heading lists its direct sub-heading OCNs. - Example output for a chapter heading: [10] heading :1 .last_descendant: 65 .children: 14 24 42 57 - Implementation: builds an int[][int] map (parent_ocn -> child heading OCNs) from one pass over the body objects, then emits .children: during serialization for headings that have entries in the map. - The tree was already reconstructable from parent_ocn + last_descendant_ocn, but .children makes it immediate - no scanning required to find a heading's sub-structure. - Tested against all 35 sample documents - zero failures. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* .ssp serializer: include all ObjGenericComposite fieldsRalph Amissah4 days1-9/+120
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Make the .ssp format a complete representation of the document abstraction by serializing all remaining fields from ObjGenericComposite (only omitting ptr.* runtime indices which are meaningless outside the in-memory context). - New fields added: .ancestors_collapsed: - collapsed level ancestor chain .dom_status: - DOM structure markedup tags status[8] .dom_status_collapsed: - DOM structure collapsed status[8] .heading_lev_collapsed: - collapsed heading level .parent_lev: - parent heading level (markup) .o_n_type: - object numbering type (0=ocn, 1=non, 2=bkidx) .is_of_type: - para/block type classification .attrib: - general attributes string .meta_lang: - block language (group/block/quote) .meta_syntax: - codeblock syntax from metainfo .sha256: - hex-encoded SHA-256 digest of object content .has: images_no_dim - image without dimensions flag .table_aligns: - column alignment array .table_walls: - table walls/borders flag .stow_link: - extracted URLs (one per line) .heading_lev_anchor: - heading level anchor tag .segment_epub: - EPUB segment anchor tag .heading_ancestors_text: - pipe-separated ancestor headings .lev4_subtoc: - sub-table-of-contents entries (one per line) .anchor_tag: - additional anchor tags (one per line) - Tested against all 35 sample documents - zero failures. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* .ssp serializer: omit identifier when it equals OCNRalph Amissah4 days1-3/+6
| | | | | | | | | | | | | | | - For heading objects, the identifier was always emitted on the declaration line (e.g. "[10] heading :1 10") even when it was just the OCN repeated. Now only emits the identifier when it differs from the OCN (i.e. when there is a named segment like "acknowledgments" or "a1"), reducing redundancy. Before: [10] heading :1 10 After: [10] heading :1 Named segments still appear: [0] heading :1 a1 Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* include .ssp document abstraction in source podRalph Amissah4 days3-1/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - When --source/--pod is used, automatically generate the .ssp document abstraction and bundle it into the pod at media/abstraction/{doc_uid}.{lang}.ssp - This makes show_abstraction implicitly true when source_or_pod is active, so the .ssp file is generated before the pod assembler runs (abstraction runs before outputHub, and source_or_pod is the first task in outputHub). - Changes: paths_source.d: Add abstraction_root() path helper to _PodPaths struct, following the same pattern as image_root(). Produces paths like pod/media/abstraction/ for both zpod (inside zip) and filesystem_open_zpod (open directory). source_pod.d: - Create media/abstraction/ directory in podArchive_directory_tree - Bundle .ssp file in pod_zipMakeReady: reads from the abstraction output directory, copies to open pod directory, adds to zip archive, computes SHA-256 digest - Write .ssp digest in zipArchiveDigest alongside sstm and ssi digests spine.d: Make show_abstraction() return true when source_or_pod is active (previously only returned true for explicit --show-abstraction flag). - The .ssp is always included when building pods - no exclusion flag for this experimental feature to keep things simple. Not generated for non-pod outputs (--text, --html, etc.) unless --show-abstraction is explicitly passed. - Tested against all 35 sample documents - zero failures. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* document abstraction as per document sqlite dbRalph Amissah4 days2-2/+388
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | --show-abstraction-db flag to write per-document - SQLite database of document abstraction (Claude-Code primary assist) - Add a new output mode that serializes the in-memory document abstraction to a per-document SQLite database. This complements the .ssp text format (--show-abstraction) with a queryable database representation of the same data. - Schema: metadata table - key/value pairs for document metadata (title, creator, dates, rights, classify, identifiers, language, notes, make settings, doc_has counts) objects table - one row per document object with columns: section, seq (position within section), ocn, is_a, is_of_part, is_of_type, heading_level, identifier, parent_ocn, last_descendant_ocn, ancestors, indent/bullet/lang, has_* flags, segment/anchor tags, table/code properties, text content Indexed on: section, ocn, parent_ocn, is_a, heading_level - Uses prepared statements via d2sqlite3 (existing dependency) for safe and efficient insertion. Each document produces a standalone .abstraction.db file in the abstraction/ output directory. - New files: src/sisudoc/io_out/create_abstraction_db.d Follows the same pattern as create_abstraction_txt.d. Creates schema, populates metadata via key/value inserts, then iterates all sections writing objects with prepared statements within a single transaction. - Changes to spine.d: - Add "show-abstraction-db" to opts init, getopt, OptActions - Add to abstraction(), require_processing_files(), and meta_processing_general() gates - Insert call at both spineAbstraction sites - Tested against all 35 sample documents (including 9-language live-manual) - zero failures. Works standalone or combined with --show-abstraction and other output flags. - Example queries the database supports: SELECT ocn, heading_level, text FROM objects WHERE is_a = 'heading' AND section = 'body'; SELECT * FROM objects WHERE parent_ocn = 10; SELECT key, value FROM metadata WHERE key LIKE 'title.%'; Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* .ssp document abstraction as PEG parsable textRalph Amissah4 days2-0/+335
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | --show-abstraction flag to write .ssp document abstraction files - Add a new output mode that serializes the in-memory document abstraction (produced by spineAbstraction) to a human-readable, line-oriented text format (.ssp). This captures the full object model after parsing and abstraction but before output generation. - The .ssp format uses unambiguous line prefixes: @section { } - section boundaries (head/toc/body/endnotes/...) [N] type - object declaration with OCN .name: value - object properties (only non-defaults) | content - text content lines % comment - comments - New files: src/sisudoc/io_out/create_abstraction_txt.d Serializer module following the same template pattern as metadoc_show_summary.d. Walks doc.abstraction() section by section, writing metadata preamble (@meta, @make, @doc_has) then each object with its properties and text content. Output goes to {output_path}/{lang}/abstraction/{doc}.ssp - Changes to spine.d: - Add "show-abstraction" to opts initialization, getopt, and OptActions struct - Add show_abstraction to abstraction(), require_processing_files(), and meta_processing_general() so the flag triggers full document processing - Insert call at both spineAbstraction sites (parallel and serial branches), gated by show_abstraction flag, following the same pattern as show_config/show_summary/show_make - Tested against all 35 sample documents (including multilingual live-manual in 9 languages) - zero failures. Works standalone (--show-abstraction) or combined with other output flags (--show-abstraction --html --text). No effect on existing code paths when the flag is not used. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* 0.19.0sisudoc-spine_v0.19.0-devRalph Amissah4 days1-1/+1
|
* upkeep, update a few pathssisudoc-spine_v0.18.0Ralph Amissah4 days11-56/+75
|
* spine may be run against a zipped spine-pod urlRalph Amissah13 days2-2/+147
| | | | | | - claude contributed src - processes zip from url using (system installed) curl for download
* spine may be run against a document-markup zip podRalph Amissah13 days2-2/+461
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | - claude contributed src - Opens the zip with std.zip.ZipArchive (reads the whole file into memory) - Locates pod.manifest inside the archive to discover document paths and languages - Extracts markup files (.sst/.ssm/.ssi) as in-memory strings - Extracts images as in-memory byte arrays - Extracts conf/dr_document_make if present - Presents these to the existing pipeline as if they were read from the filesystem - Some security mitigations: - Zip Slip / Path Traversal: Reject entries containing `..` or starting with `/`; canonicalize resolved paths and verify they fall within extraction root - Zip Bomb: Check `ArchiveMember.size` before extracting; enforce per-file (50MB) and total size limits (500MB) - Entry Count: Limit number of entries (a pod should have at most ~100 files) - Path depth: limit (Maximum 10 path components). - Symlinks: Verify no symlinks in extracted content before processing (post-extraction recursive scan) - Filename Validation: Only allow expected characters; reject null bytes - Malformed Zips: Catch `ZipException` from `std.zip.ZipArchive` constructor - Cleanup on error
* package.nix cosmetic line-breaks for build commandRalph Amissah13 days1-1/+4
|
* nix keeping: nix-shell, nix-build derivation.nixRalph Amissah2026-04-112-102/+130
|
* flake.nix dmd build fix overlay: revert to GCC14Ralph Amissah2026-04-093-67/+14
| | | | | | | | | - revert to using GCC14: (current) GCC 15 introduced nullptr in its headers, and DMD's ImportC parser needs update to handle it, monitor and update - (nix ldc overlay, minor comsetic)
* nix cleanup flake.nix and shell.nixRalph Amissah2026-04-071-391/+191
|
* latex minor improvements and fixes, require testingRalph Amissah2026-04-061-9/+12
| | | | | - FIXES issue with .tex files and xetex finding image paths when run within latex/ output directory
* package housekeeping, minorRalph Amissah2026-03-193-8/+36
|
* nix direnv 3.1.0Ralph Amissah2026-03-051-2/+15
|
* ldc-1.42.0 overlayRalph Amissah2026-03-011-19/+3
|
* ldc-1.42.0-beta2 overlayRalph Amissah2026-02-051-3/+17
|
* ldc-1.42.0-beta1 overlayRalph Amissah2026-02-052-158/+28
|
* org-mode header, minor (set: results silent)Ralph Amissah2026-01-1043-44/+44
|
* 2026Ralph Amissah2026-01-0944-52/+205
|
* flake.nix po4a perl depsRalph Amissah2026-01-081-2/+2
|
* envrc markup samples pod dir changeRalph Amissah2025-10-311-1/+2
| | | | - markup samples pod dir: markup/pod-samples/pod
* nix flake minor housekeepingRalph Amissah2025-10-311-15/+19
|
* envrc minor, (including emacs syntax highlighting)Ralph Amissah2025-10-311-4/+6
|
* text output, improve various (including no-ocn)Ralph Amissah2025-10-143-36/+34
| | | | - revisit links (fix later)
* abstraction metainfo, provide endnote parent ocnRalph Amissah2025-10-133-11/+7
| | | | | | - preferable, endnote parent object number available for use (as here in text output, compare "endnotes, add caller ocn" commit)
* housekeeping, nix & env, minorRalph Amissah2025-10-132-1/+34
|
* ,version (shellscript: project & git version info)Ralph Amissah2025-10-092-0/+14
|
* houskeeping, (emacs org tangle updates), minorRalph Amissah2025-10-082-2/+4
|
* latex quote object, quick fixRalph Amissah2025-10-081-2/+29
|
* text output, endnotes, add caller ocn (& some cleaning)Ralph Amissah2025-10-083-13/+33
|
* a text output (and skel an outline)Ralph Amissah2025-10-0310-33/+1053
| | | | - spine --text [--output=output path] [markup source]
* (editor emacs org-mode includes)Ralph Amissah2025-09-2631-78/+596
|
* terminal output verbosity levels, minor reworkRalph Amissah2025-09-2512-191/+223
|
* spine.d tidyRalph Amissah2025-09-231-109/+88
|
* dub_describe.json + other minor miscRalph Amissah2025-09-143-52/+50
|
* org, property header-args results output noneRalph Amissah2025-09-1441-42/+42
|
* dub 1.40.0, llvm 20Ralph Amissah2025-08-082-47/+68
|
* imports, make line searchableRalph Amissah2025-07-1518-407/+344
|
* source & pod (fix build from non-pod source)Ralph Amissah2025-06-122-23/+26
| | | | - appears to work, but needs review
* nix flake and env updates, ldc using llvm-20Ralph Amissah2025-06-125-37/+33
| | | | - .env/ removed as unused & unmaintained
* ldc-1.41.0 overlay; nix direnv 3.0.7Ralph Amissah2025-06-071-11/+69
| | | | | - related updates to files - notes on updating these added (.org)
* org ready ldc-1.41.0-beta1; flake using ldc-1.40.1Ralph Amissah2025-04-185-27/+57
| | | | - plus minor housekeeping/tidy
* dmd-2.111.0Ralph Amissah2025-04-022-12/+14
|
* minorRalph Amissah2025-04-021-1/+0
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-26 16:19:21 +0000