| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The segment a cross reference to a heading lands on, gixed upstream in
ocda, where the field is created, rather than inferred by the loader.
A heading above level 4 opens no html segment of its own, so a link to
it has to land on the level 4 heading that follows. The build loop
cannot know that when it reads the heading, and says so in a comment of
its own: "for html segname need following lv4 not yet known". It
back-fills tag_assoc when the level 4 heading arrives (lv0to3_tags), and
the answer then lives only in that map.
tags.segment_lv4_is now holds it, resolved in a pass over the finished
head and body sections: walking backwards, the last level 4 heading seen
is the next one for everything above it, which is the same answer the
back-fill gives. Emitted in the .ssp only where it differs from
.segment_html_is, which is every heading at level 4 or below, so it is
sparse: 317 lines over the 35 document reference set, none removed.
Carried in the database as a column of its own, and read back by both
readers.
docHasFromAbstraction reads it instead of working it out. With that, and
on top of the two defect fixes, a document loaded from an artefact and
one parsed from markup agree:
key sets identical, all 35 documents
values 3 entries differ of some 30,000, all of them
_the_title, where the abstraction supplies an epub
segment the parser leaves unset
link targets all 1,651 agree, against 89 differing before any
of this work and 1 after the defect fixes alone
Format stays v1.1. That version is new in this same run of work and
nothing outside spine has read it, so this belongs in it rather than in
a bump of its own.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
.ssp format v0.1 -> v1.1, and the .ocda.db with it: they are two
serialisations of one format and now move together.
The gap this closes, measured against the writers' read set: of the 59
conf_make_meta fields outputs/io_out/ reads, 29 were in neither
artefact. Five are conf.* and stay out, being site and run scoped (urls,
papersize, the search db filename): the same abstraction published to
two sites must take each site's. Of the rest, three are never assigned
anywhere (title_short, publisher, original_publisher; read only by
sqlite.d, always empty) and one is a copy of a field already carried
(title_sub = title_subtitle), so 17 properties actually had to travel
and now do:
make breaks, footer, home_button_text
meta title.edition, date.added_to_site, language.document_char,
original.{title,source,language,language_char},
rights.copyright_{text,translation,illustrations,
photographs,cover,audio,video}
make splits by when it acts, which is worth keeping in mind: these three
are read in outputs/io_out/ and must travel, while italics, bold,
emphasis, substitute and headings are read at parse time and their
effect is already in the objects.
New @source block, and source.* rows in the database, so a reader can
say which markup an abstraction came from rather than working from
stale content in silence:
language the document's own
languages the pod's list, which is what the inter-language links
in html need and neither artefact carried
digest sha256 of the .sst; equals its digests.txt entry
The database adds source.ssp_digest, the sha256 of the .ssp it was built
from, since a file cannot hold its own hash. So the chain
.sst -> .ssp -> .ocda.db is checkable end to end.
The database's metadata table is now filled from the header blocks the
.ssp gives back, not from a second list read off doc_matters. One list,
in ssp.d: a property added there arrives in the database with nothing
else changed, and one that is not in the .ssp cannot be in the database
at all. That was the last place the two could drift; the objects stopped
being able to on 2026-09-07.
Version is checked on load. The major part must match, a newer minor is
accepted (a minor bump only adds properties, and an unknown property
line is ignored). A v0.1 artefact is now refused with a message saying
to regenerate it, rather than loading half populated.
test-abstraction-db.sh now compares the two artefacts' header blocks
property by property, 36 per document on the wealth of networks, where
it previously only checked that a schema.version row existed. Verified
non-vacuous: dropping one property is reported.
Reference regenerated, and the whole diff is this change and nothing
else: 320 lines added, 35 removed over 35 files, being 35 format lines
changed, 35 @source blocks (4 lines each), 35 language.document_char, 35
home_button_text (it has a default), 18 breaks, 16 footer,
4 date.added_to_site, 1 title.edition, 1 original.source.
Existing outputs unaffected.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
| |
The abstraction has nine sections and includes "tail". Both artefact
writers hardcoded a list of eight, so "tail" was silently dropped.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
| |
serial processing, it turns out, is significantly faster and more
efficient for tested use-cases, which came as a surprise. As the
parallelization option buys nothing, serial processing is set as
default. Parallel processing remains as an option (where
available, as before).
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix issue with consistency (flags run serial & parallel
inconsistenly).
Both write one file per document per language and share no
handle, so they belong on the list with html, epub, text and
sqlite_discrete.
The guard above the list still takes out --pod, --pod2,
--source and the shared sqlite db actions before it is reached, so
those stay serial as they were; checked.
However measurement tests show parallel turn out to result in a
processing slowdown, on the (35) sample markup documents, run on
multiple passes: run on 16 cores a slow down of about 20% for
eleven times the cpu!
--text 4.83-5.10 s wall 48-52 s user
--text --serial 3.94-4.05 s wall 4.2-4.4 s user
will make serial run the default.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
fix of a couple of issues with tests.
test-search-cgi.sh reported PASS while printing "Schema version gate:
skipped, sqlite3 not on PATH", and the values it searches for were the
hardcoded fallbacks rather than values read out of the database it had
just built. Both halves need sqlite3, both are checks, and a run that
leaves them out still exits 0. The other three scripts re-run
themselves under nix shell "nixpkgs#sqlite"; this one now does the
same. sqlite3 is still not required: with neither sqlite3 nor nix the
test runs as before and says what it skipped.
test-abstraction-ssp.sh depends on --abstraction for its meaning, and
nothing in it said so. --show-abstraction does not appear in spine's
parallelise() list, so dropping that one flag turns the current run
serial, and the parallel-versus-serial comparison below it then
compares a serial run with itself - the guard gone, with nothing
failing to report it. That is how the multi-lingual write race survived.
A comment at the call now says not to drop it.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Spine now declares sqlite_db_schema_version and stamps it into the
database as PRAGMA user_version when the tables are created, in
both the shared and the discrete DDL blocks. On opening an
existing database it compares, and says once per run which version
it found and which it writes.
Failures are now tallied (shared, the output can run in parallel),
reported one line each on stderr naming the operation, and main
exits 1 without printing "run complete, ok".
Two tests under test/, both taking the spine binary as their first
argument and building their own database from data/pod unless
$SpinePOD says otherwise:
test-search-db-schema.sh names: every column the search form
uses exists, and spine's declaration, the database's stamp
and the search form's expectation all agree
test-search-cgi.sh behaviour: the real search binary answers
real requests against a fresh database, no web server
involved, the probe values read out of whichever database it
is given
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
| |
ocda (object centric document abstraction)
flag --ocda-db (or --abstraction-db)
replaces --show-abstraction-db
rename results in consequently large diff
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pod/ holds all document source representations:
pod/<doc>/ source tree
pod/<doc>/media/abstraction/<uid>.ssp abstraction, as text
pod/<doc>.zip tree, zipped, .ssp included
pod/<doc>.digests.txt sha256s of what is in them
pod/<uid>.ocda.db abstraction, as sqlite db
<lang>/abstraction/ is gone.
pod/<doc>/media/abstraction/<uid>.ssp preferred as having the
images (found within the pod tree) which .ssp needs to reproduce a
document but does not carry on its own.
The .ocda.db sits carries the images as well and (like the
pod.zip) can be used to reproduce a document directly.
digests.txt now covers the database as well as the zip, the source
and the .ssp; (as does the metadata html page).
Two ordering issues addressed:
- the pod builder clean-slates pod/<doc>/ before regenerating it,
and the .ssp is now written before that runs. The clean slate
now leaves media/abstraction/ alone, and the .ssp writer clears
that directory itself on the first language of a run, so a .ssp
for a language the document no longer is removed and cannot be
bundled.
- for a multi-language document the .ssp files accumulate one
language at a time and are bundled on the last, which is why the
directory cannot simply be emptied by whichever (language) gets
there first.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The per document database is now written as <doc_uid>.ocda.db
rather than <doc_uid>.abstraction.db. Shorter, and it says what is
in the file: the object centric document abstraction, not "some
abstraction". .ocda.db pairs with .ssp and cannot be mistaken for
the collection search database (spine.search.db).
test/run-tests.sh runs the four in sequence, one line of result
each, and a summary. It re-runs itself inside nix shell
"nixpkgs#sqlite" if sqlite3 is not on PATH, so this is all that
is needed:
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
./test/run-tests.sh ./bin/spine-ldc
The tests are independent (with test-abstraction-ssp.sh run
first):
1 test-abstraction-ssp.sh is first because it is the
one that says whether the abstraction itself moved; if it
fails the others are answering a different question than you
think
2 test-abstraction-ssp-roundtrip.sh reads the committed
reference set, so it is a statement about the current binary
only if 1 passes
3 test-abstraction-db.sh and
4 test-abstraction-db-roundtrip.sh generate both artefacts
themselves and depend on nothing committed
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sisudoc.ocda.abstraction.db_in reads a <doc>.abstraction.db back
into ObjGenericComposite[][string], the same value ssp_in returns
from a .ssp, so a consumer need not know which artefact it was
handed. It mixes in the .ssp reader for that shared document
struct rather than declaring a second one.
--db-round-trip=<file.abstraction.db> reads a database and emits
it as .ssp on stdout, through sspObjectRecord as the other round
trip does. Held against the .ssp written from the same document,
this says whether the two artefacts really carry the same thing:
not a count of fields, as test-abstraction-db.sh does, but the
whole document reconstructed from the database and compared to the
text.
SpinePOD=... ./test/test-abstraction-db-roundtrip.sh ./bin/spine-ldc
PASS: all 35 databases re-emit their document's .ssp exactly
It passed on the first run over the whole sample set, which is evidence
that the database is now field-complete against the .ssp rather than
merely counting the same.
Four tests with different checks:
test-abstraction-ssp.sh the abstraction has not changed
test-abstraction-db.sh the two serialisations agree, field by field
test-abstraction-ssp-roundtrip.sh the .ssp can be read back whole
test-abstraction-db-roundtrip.sh the .db can be read back whole
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reads each committed reference .ssp back into the abstraction, emits
it again, and requires byte identity. Both directions go through
template sspObjectRecord, the writer's own definition of a record, so
what is being checked is the reader against the writer and not against
a second description of the format.
./test/test-abstraction-ssp-roundtrip.sh ./bin/spine-ldc
PASS: all 35 .ssp files read back and re-emit byte identically
It needs no sample documents and no output directory; the reference
set is the input. So the three tests now say three different things:
test-abstraction-ssp.sh the abstraction has not changed
test-abstraction-db.sh the two serialisations agree
test-abstraction-ssp-roundtrip.sh the .ssp can be read back whole
Added with -f, as test/ is covered by .gitignore's catch-all.
(assisted by Claude-Code)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With these the .ssp carries every field the output writers read.
The two that were missing were both read by the sqlite search
writer, and one of them is needed by anything that wants to know
where an object sits in a segmented output:
- .segment_html_is / .segment_epub_is
(tags.html_segment_anchor_tag_is, epub_segment_anchor_tag_is),
the segment an object *belongs to*, as against .segment
(in_segment_html), which is the segment a heading *opens* and is
set on headings only: 32 of 36 headings and 0 of 109 paragraphs
in the autonomous contract. Without them a reader cannot tell
which segment a paragraph is in.
- .identifier for non-heading objects, the "a"~N series given to
objects whose number is suppressed. A heading already carried it
on its declaration line; every other object now carries it as a
property, on the same condition, that it differs from the ocn.
The same two segment fields are added to the abstraction db
(segment_html_is, segment_epub_is) and to the cross-check test, so the
two serialisations stay level.
.ssp reference regenerated
output run (--text --latex --epub --html --curate) byte identical
to before the change, as it must be since no output writer reads
either artefact.
(assisted by Claude-Code)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
cross-check .ssp & ocda (abstraction) db against each other
To prevent drift, this compares them directly to detect one writer
gaining a field and the other not (rather than each against a
committed reference, so there is no new fixture to keep current)
For every sample document it generates both artefacts and asserts:
- the same number of objects, and the same number in each of the eight
sections
- for 32 single valued properties, that the count of the .ssp property
equals the count of rows where the corresponding column is set
- for the four repeatable properties (.image, .stow_link, .anchor_tag,
.lev4_subtoc), that the count equals the rows in its table
- that every image the abstraction referred to and found is carried in
the files table
- that the text of the first and last citable body object is byte
identical in the two
- that the database says what it is (metadata schema.version)
SpinePOD=../../markup/sisudoc-spine-samples/markup/pod-samples/pod \
nix shell "nixpkgs#sqlite" -c ./test/test-abstraction-db.sh ./bin/spine-ldc
A sqlite error on a missing column is reported as a mismatch
rather than aborting the run.
The other half of this step, one shared field description behind
both writers, is not done here: the cross-check gives the same
guarantee without touching either writer, and the question becomes
moot if, as planned, the database comes to be generated from the
.ssp through the loader.
Added with -f: test/ is covered by .gitignore's catch-all, as
test/test-abstraction-ssp.sh was before it.
(assisted by Claude-Code)
|
| |
|
|
| |
track on sisu markup sample
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Several structure fields (now) computed in ocda do not yet have a
consumer outside the `.ssp` writer (which shares ocda fields in a
PEG parsable .ssp file). Their addition is to facilitate future
output processing: each representing document structure
information that document output generators of various formats,
such as html, epub, latex, sql and any DOM based / tree-graph
(node based hierarchical) representations, previously had to, or
would have to, work out for themselves based on what ocda offered
(without them).
Computing them (upstream) in ocda once and sharing them (and
providing the .ssp), means that:
- future output formats can start from prepared structure rather
than deriving it;
- existing generators can be simplified to consume them instead
of re-deriving;
- a (dlang) generator could in principle be populated either from
ocda directly or from a `.ssp` file;
- `.ssp` being PEG parsable puts the same prepared structure in
reach of implementations in other languages.
`test/test-abstraction-ssp.sh` the `.ssp` reference regression
test is used to catch regression in ocda including these as yet
unused fields, which otherwise would remain unseen until used by
other output-format consumers. This also is the reason issues now
made easily visible and are being addressed (without issue to
existing output generators).
(assisted by Claude-Code)
|
| |
|
|
|
| |
test-abstraction-ssp re-generated: document abstraction (ocda)
document structure representation fixes (verify)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
for document abstraction and its .ssp output, removed the
requirement of including --abstraction & --serial flags to produce
correct output (for: .dom_status, .dom_status_collapsed &
.last_descendant)
- meta_processing_xml_dom() includes show_abstraction, so --pod2
and --show-abstraction run the dom pass; last_descendant is
derived from that pass via after_doc_get_descendants()
The accumulators are now verified as eight wide locals of
docAbstraction(), so each document starts clean and no two threads
share one.
- bug: the four dom accumulators were template scope (shared) and
nine wide, while their end of document reset was eight wide, so
the first document of a run differed from the rest and parallel
runs raced on one buffer, (which also mis-nested epub toc_nav)
test/ reference .ssp regenerated: accelerando only, trailing zero
dropped.
test-abstraction-ssp.sh now runs parallel and diffs output against
a serial run.
(assisted by Claude-Code)
|
| | |
|
| | |
|
| |
|
|
|
| |
generrate new reference set of .ssp
test-abstraction-ssp.sh --generate
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
test-abstraction-ssp.sh should fail if $SpinePOD is unset.
SAMPLES_RAW="$SPINE_DIR/$SpinePOD" resolves to the spine directory
itself when the variable is unset. That directory exists, so the
"sample documents not found" check did not fire and spine was
handed every file in the spine tree.
Fail early instead, naming the variable and giving the path the
samples are usually at.
(assisted by Claude-Code)
|
| |
|
|
|
|
| |
corrects an omission this hould have been included to start with
reference tests texts regenerated
|
| |
|
|
|
|
|
|
|
| |
- test/test-abstraction-ssp.sh script minor improvement
- result change reflects content corrections: name spelling & a markup
error, document abstracton, ssp verified ok (may be considered a test)
(assisted by Claude-Code)
|
|
|
test/test-abstraction-ssp.sh: shell script that generates
.ssp files for all sample documents and diffs against
committed reference files in test/reference/abstraction/.
Usage:
./test/test-abstraction-ssp.sh --generate ./bin/spine-ldc
./test/test-abstraction-ssp.sh ./bin/spine-ldc
Exits 0 if all .ssp files match, 1 if differences found.
Reports changed, missing, and new files.
test/reference/abstraction/: 35 reference .ssp files
generated from the current abstraction, covering all
sample documents including 9-language live-manual.
Co-Authored-By: Anthropic Claude Opus 4.6 (1M context)
|
