aboutsummaryrefslogtreecommitdiffhomepage
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* document abstraction as per document sqlite dbRalph Amissah2026-04-222-0/+373
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | --show-abstraction-db flag to write per-document - SQLite database of document abstraction (Claude-Code primary assist) - Add a new output mode that serializes the in-memory document abstraction to a per-document SQLite database. This complements the .ssp text format (--show-abstraction) with a queryable database representation of the same data. - Schema: metadata table - key/value pairs for document metadata (title, creator, dates, rights, classify, identifiers, language, notes, make settings, doc_has counts) objects table - one row per document object with columns: section, seq (position within section), ocn, is_a, is_of_part, is_of_type, heading_level, identifier, parent_ocn, last_descendant_ocn, ancestors, indent/bullet/lang, has_* flags, segment/anchor tags, table/code properties, text content Indexed on: section, ocn, parent_ocn, is_a, heading_level - Uses prepared statements via d2sqlite3 (existing dependency) for safe and efficient insertion. Each document produces a standalone .abstraction.db file in the abstraction/ output directory. - New files: src/sisudoc/io_out/create_abstraction_db.d Follows the same pattern as create_abstraction_txt.d. Creates schema, populates metadata via key/value inserts, then iterates all sections writing objects with prepared statements within a single transaction. - Changes to spine.d: - Add "show-abstraction-db" to opts init, getopt, OptActions - Add to abstraction(), require_processing_files(), and meta_processing_general() gates - Insert call at both spineAbstraction sites - Tested against all 35 sample documents (including 9-language live-manual) - zero failures. Works standalone or combined with --show-abstraction and other output flags. - Example queries the database supports: SELECT ocn, heading_level, text FROM objects WHERE is_a = 'heading' AND section = 'body'; SELECT * FROM objects WHERE parent_ocn = 10; SELECT key, value FROM metadata WHERE key LIKE 'title.%'; Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* .ssp document abstraction as PEG parsable textRalph Amissah2026-04-222-0/+322
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | --show-abstraction flag to write .ssp document abstraction files - Add a new output mode that serializes the in-memory document abstraction (produced by spineAbstraction) to a human-readable, line-oriented text format (.ssp). This captures the full object model after parsing and abstraction but before output generation. - The .ssp format uses unambiguous line prefixes: @section { } - section boundaries (head/toc/body/endnotes/...) [N] type - object declaration with OCN .name: value - object properties (only non-defaults) | content - text content lines % comment - comments - New files: src/sisudoc/io_out/create_abstraction_txt.d Serializer module following the same template pattern as metadoc_show_summary.d. Walks doc.abstraction() section by section, writing metadata preamble (@meta, @make, @doc_has) then each object with its properties and text content. Output goes to {output_path}/{lang}/abstraction/{doc}.ssp - Changes to spine.d: - Add "show-abstraction" to opts initialization, getopt, and OptActions struct - Add show_abstraction to abstraction(), require_processing_files(), and meta_processing_general() so the flag triggers full document processing - Insert call at both spineAbstraction sites (parallel and serial branches), gated by show_abstraction flag, following the same pattern as show_config/show_summary/show_make - Tested against all 35 sample documents (including multilingual live-manual in 9 languages) - zero failures. Works standalone (--show-abstraction) or combined with other output flags (--show-abstraction --html --text). No effect on existing code paths when the flag is not used. Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
* upkeep, update a few pathssisudoc-spine_v0.18.0Ralph Amissah2026-04-221-10/+10
|
* spine may be run against a zipped spine-pod urlRalph Amissah2026-04-132-2/+147
| | | | | | - claude contributed src - processes zip from url using (system installed) curl for download
* spine may be run against a document-markup zip podRalph Amissah2026-04-132-2/+457
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | - claude contributed src - Opens the zip with std.zip.ZipArchive (reads the whole file into memory) - Locates pod.manifest inside the archive to discover document paths and languages - Extracts markup files (.sst/.ssm/.ssi) as in-memory strings - Extracts images as in-memory byte arrays - Extracts conf/dr_document_make if present - Presents these to the existing pipeline as if they were read from the filesystem - Some security mitigations: - Zip Slip / Path Traversal: Reject entries containing `..` or starting with `/`; canonicalize resolved paths and verify they fall within extraction root - Zip Bomb: Check `ArchiveMember.size` before extracting; enforce per-file (50MB) and total size limits (500MB) - Entry Count: Limit number of entries (a pod should have at most ~100 files) - Path depth: limit (Maximum 10 path components). - Symlinks: Verify no symlinks in extracted content before processing (post-extraction recursive scan) - Filename Validation: Only allow expected characters; reject null bytes - Malformed Zips: Catch `ZipException` from `std.zip.ZipArchive` constructor - Cleanup on error
* latex minor improvements and fixes, require testingRalph Amissah2026-04-061-9/+12
| | | | | - FIXES issue with .tex files and xetex finding image paths when run within latex/ output directory
* 2026Ralph Amissah2026-01-0948-49/+49
|
* text output, improve various (including no-ocn)Ralph Amissah2025-10-144-36/+35
| | | | - revisit links (fix later)
* abstraction metainfo, provide endnote parent ocnRalph Amissah2025-10-134-12/+8
| | | | | | - preferable, endnote parent object number available for use (as here in text output, compare "endnotes, add caller ocn" commit)
* latex quote object, quick fixRalph Amissah2025-10-081-2/+22
|
* text output, endnotes, add caller ocn (& some cleaning)Ralph Amissah2025-10-084-25/+45
|
* a text output (and skel an outline)Ralph Amissah2025-10-0310-41/+885
| | | | - spine --text [--output=output path] [markup source]
* terminal output verbosity levels, minor reworkRalph Amissah2025-09-2514-107/+130
|
* spine.d tidyRalph Amissah2025-09-231-109/+88
|
* dub_describe.json + other minor miscRalph Amissah2025-09-142-49/+49
|
* src/ext_deplends d-yaml updated (v0.10.0)Ralph Amissah2025-08-28687-5103/+7987
|
* imports, make line searchableRalph Amissah2025-07-1529-432/+359
|
* source & pod (fix build from non-pod source)Ralph Amissah2025-06-122-23/+26
| | | | - appears to work, but needs review
* org ready ldc-1.41.0-beta1; flake using ldc-1.40.1Ralph Amissah2025-04-181-2/+2
| | | | - plus minor housekeeping/tidy
* minorRalph Amissah2025-04-021-1/+0
|
* sisudoc-spine upkeep, minor, a file renamedRalph Amissah2025-03-222-0/+1
|
* triple single-quote marks block identifier addedRalph Amissah2025-02-214-4/+229
| | | | | | | | - tics a bit cumbersome where single quotes work just as well - testing required (special cases not covered) - diverges from sisu markup which will need an update sometime
* doc (metadata & abstraction) struct follow throughRalph Amissah2025-02-195-275/+236
|
* document (metadata & abstraction) structRalph Amissah2025-02-198-107/+100
| | | | | | - struct replaces tuple - some direct naming of structs returned (instead of use of auto) - minor
* 2025Ralph Amissah2025-01-0146-47/+47
|
* refactor yaml extraction code fileRalph Amissah2024-12-212-671/+354
|
* nix build flake.nix fixRalph Amissah2024-12-032-0/+2
|
* pod zip fixesRalph Amissah2024-07-104-131/+120
| | | | | - serial processing (need to be built serially) - multilingual pods, copy all languages before zip
* [fn].digest.txt, sha256 of pod source files & podRalph Amissah2024-07-047-339/+379
|
* markup source digest to metadata.htmlRalph Amissah2024-07-011-3/+18
|
* markup source digests (write to terminal)Ralph Amissah2024-07-016-9/+30
|
* digest tuple rearrangeRalph Amissah2024-06-293-21/+27
|
* reduction in use of tuplesRalph Amissah2024-06-291-67/+70
|
* document digests and reduction in use of tuplesRalph Amissah2024-06-297-46/+83
|
* latex footers from document header make, a fixRalph Amissah2024-05-293-14/+17
|
* update fix default home text link infoRalph Amissah2024-05-171-6/+6
| | | | - used e.g. in html text home button
* dub (dlang) prefer dub run to dub buildRalph Amissah2024-05-072-28/+0
|
* 0.16.0 sisudoc (src/sisudoc sisudoc spine)sisudoc-spine_v0.16.0-devRalph Amissah2024-04-1047-308/+308
| | | | | - src/sisudoc (replaces src/doc_reform) - sisudoc spine (used more)
* 0.15.0doc-reform_v0.15.0Ralph Amissah2024-04-1037-74/+74
|
* mark modules as @safe: (& identify what is not)Ralph Amissah2024-03-1240-341/+380
|
* org, ocda (ongoing) split file, separate functionsRalph Amissah2024-01-254-5408/+5526
|
* org, ocda (ongoing)Ralph Amissah2024-01-211-1641/+1558
|
* org, ocda (ongoing)Ralph Amissah2024-01-021-26/+13
|
* 2024Ralph Amissah2024-01-0245-46/+46
|
* org, ocda (ongoing)Ralph Amissah2023-12-211-274/+289
|
* org, ocda (ongoing)Ralph Amissah2023-12-171-8/+27
|
* org, spine ver & doc header (copyright & license)Ralph Amissah2023-12-173-6/+4
| | | | | | - org import: reuse settings in another org file using code block to evaluate other org file spine_version_info_and_doc_header_including_copyright_and_license.org
* ocda, structs (replacing tuples)doc-reform_v0.13.0Ralph Amissah2023-11-191-233/+238
|
* ocda, minor text formatting for comments & debugsRalph Amissah2023-11-191-565/+244
| | | | | | - side-note: issues with doom emacs org babel tangle (not completely resolved, and have persisted)
* ocda, arrange structs, continueRalph Amissah2023-11-161-188/+189
|
generated by cgit v1.2.3 (git 2.25.1) at 2026-05-22 07:23:06 +0000