diff options
Diffstat (limited to 'README')
| -rw-r--r-- | README | 527 |
1 files changed, 296 insertions, 231 deletions
@@ -15,41 +15,47 @@ #+PROPERTY: header-args+ :cache no #+PROPERTY: header-args+ :padline no -project_name: Spine, Doc Reform +project_name: "sisudoc spine (doc reform)" - description: [ - "documents, structuring, processing, publishing", - search, - object numbering, - static content generator, - sisu markup - ] +description: + - "documents, structuring, processing, publishing" + - "search" + - "object numbering" + - "static content generator" + - "sisu markup" - author: - name: Ralph Amissah - email: ralph.amissah@gmail.com +author: + name: "Ralph Amissah" + email: ralph.amissah@gmail.com - copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved." +copyright: "(C) 2015 - 2024 Ralph Amissah, All Rights Reserved." - license: "AGPL 3 or later" +license: + - "project code: AGPL 3 or later" - homepage: [ - "https://www.sisudoc.org", - "https://www.doc-reform.org" - ] +homepage: + - "https://sisudoc.org" + - "https://doc-reform.org" -* Installation, Compilation +git: + - "https://git.sisudoc.org" -Development of sisudoc-spine started in 2015 on a Debian linux box. +* Summary -Development since 2020 has been on a NixOS linux box, my laptop. If you are -fortunate enough to be using the same the build instructions should be presented -on entering the sisudoc-spine directory. It should be little problem building on -other linuxes with the right dependencies. At one time, debconf-18 I was -persuaded to try meson, and for a couple of years maintained a meson build, that -dropped out of use before or on my making the switch to nixos in 2020. +SiSU is an object-centric, lightweight markup based, document structuring, +parser, publishing and search tool for document collections. It is command line +oriented and generates static content that is currently made searchable at an +object level through an SQL database. Markup helps define (delineate) objects +(primarily various types of text block) which are tracked in sequence, +substantive objects being numbered sequentially by the program for object +citation. -❯❯ D compiler and build manager +Development of sisudoc-spine started in 2015 on a Debian linux box as a +replacement for sisu (written in Ruby, starting 2000, and Perl from 1997). +(Using Nix and NixOS since 2020). + +* Compilation, Installation +** D compiler (dmd, ldc2) & D build manager (dub) SiSU spine is written in the programming language D for which there are 3 compilers: dmd, ldc, gdc @@ -61,7 +67,9 @@ D projects tend to use dub as project manager The default build tools used are dub with ldc2 (dub is also tested) -** make a directory and clone the sisudoc-spine project +** Clone project + +Make a directory and clone the sisudoc-spine project mkdir ~/git.sisudoc cd ~/git.sisudoc @@ -82,125 +90,120 @@ sisudoc-spine cd sisudoc-spine -## directly with dub -### ldc2 +to build directly with dub, either: + +for ldc2: # on nix (get dependencies by setting your development environment): nix develop ".#dsh-nixpkgs-ldc-dub" --print-build-logs -c zsh + # assuming you have ldc2 & dub installed on your system: dub run --compiler=ldmd2 --config=ldmd2 --combined --skip-registry=all dub --compiler=ldmd2 --config=ldmd2 dub run --compiler=ldc2 --config=ldc2 --combined --skip-registry=all dub --compiler=ldc2 --config=ldc2 -### dmd +for dmd: # on nix (get dependencies by setting your development environment): nix develop ".#dsh-nixpkgs-dmd-dub" --print-build-logs -c zsh + # assuming you have dmd & dub installed on your system: dub run --compiler=dmd --config=dmd --combined --skip-registry=all dub --compiler=dmd --config=dmd -## with make +to build with make using the provided makefile, (assuming you have the named +compiler and dub installed on your system) either: -### ldc2 +for ldc2: make ldc -### dmd +for dmd: make dmd -## with nix on linux / nixos +to build using nix flakes on linux / nixos -### ldc2 +for ldc2: nix build ".#spine-nixpkgs-ldc" --print-build-logs -### dmd +for dmd: nix build ".#spine-nixpkgs-dmd" --print-build-logs -## the Meson build system was used briefly - -On recommendation at debconf-18 meson was used briefly. It has neither been tested nor used since the move to nix. - -- https://mesonbuild.com/ +The Meson build system was used briefly to build spine, but the spine build +tooling for Meson has not been updated, maintained or tested in recent years. meson ninja -C build meson setup --wipe build && ninja -v -C build make meson +- https://mesonbuild.com/ + dub --force --compiler=ldc2 && sudo cp -v cgi-bin/spine-search /usr/lib/cgi-bin/. -* Document processing examples +* Commands - document processing examples -These examples assume the file layout suggested in cloning the git.sisudoc.org -repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples -are next to each other on a directory tree. Assuming this to be the case, you -may wish to set the following exports with adjustments accoring to your specific -needs for these examples. +** basic output -# ❯❯ set spine binary location: -export SpineBIN=./result/bin/spine -# ❯❯ nix builds spine binary: -#export SpineBIN=./result/bin/spine -# ❯❯ dub builds spine binary (name depends on build, check): -#export SpineBIN=./bin/spine -#export SpineBIN=./bin/spine-ldc -#export SpineBIN=./bin/spine-dmd -# ❯❯ location of source files: -export SpineDOC=../sisudoc-spine-samples -# ❯❯ location of source files pod: -export SpinePOD=${SpineDOC}/markup/pod -# ❯❯ sisudoc-spine output processing path: -export SpineOUT=./OUTPUT_TEST_sisudocSpine -# ❯❯ sisudoc-spine output processing path (web server e.g.): -#export SpineOUT=/srv/www/spine -export SpineSearchActionLocal='http://localhost/spine_search' -export SpineSearchActionRemote='https://sisudoc.org/spine_search' -# ❯❯ path configured for cgi search form: -export SpineCGIform='spine_search' -# ❯❯ search form db name: -export SpineSQLdb='spine.search.db' -# ❯❯ configuration cgi search form path: -#export SpineCGIbin=/var/www/cgi/cgi-bin -# ❯❯ configuration db path: -#export SpineDBpath=/var/www/sqlite +For the most basic output you will need to specify: -*** html with links to search form +- the spine binary (executable) +- the (recognized) path to a prepared (spine marked up) document or document + collection +- the (path to) where the output is to be placed +- the output types you seek -${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/* +export SpineBIN=./result/bin/spine +export SpinePOD=../sisudoc-spine-samples/markup/pod +export SpineOUT=./OUTPUT_TEST_sisudocSpine +${SpineBIN} -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/* ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/* -${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/* +which would execute the following command: -spine -v --html \ - --html-link-search \ - --output=`echo ~webDocRoot` \ - ${SpinePOD}/* +./result/bin/spine -v --source --pod --epub --html --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* +./result/bin/spine -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* ** curate if you have a document collection with documents that have metadata headers a -summary of the collection can be made using the curate command +summary of the collection can be made using the curate command: - spine -v --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* +${SpineBIN} -v --curate --output=${SpineOUT} ${SpinePOD}/* - spine -v --curate ~spineMarkupSamples/pod/* +spine -v --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* - spine -v --html --html-link-search --html-link-curate --curate --output=`echo ~webDocRoot` ~spineMarkupSamples/pod/* +${SpineBIN} -v --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/* - spine -v --html --html-link-search --html-link-curate --curate ~spineMarkupSamples/pod/* +spine -v --html --html-link-curate --curate --output=./OUTPUT_TEST_sisudocSpine ../sisudoc-spine-samples/markup/pod/* ** sqlite +Configuration and setup are required to use sqlite search with sisudoc-spine for +the first. + +- sqlite3 will need to be installed and recognized as such by the program + +- you will need to have a web server configured to run cgi + +- sisudoc-spine-search-cgi will need to be compiled and the binary placed in the + appropriate cgi path + +- you will need to use sisudoc-spine to initialize the database (create tables + and indexes) + +- sisudoc-spine can be used to populate the database, and produce html with + entry submission fields that link to the cgi search + if configuartion has been set specify just - the desired output and - the markup document/pod(s) to process - spine -v --html ~spineMarkupSamples/markup/pod/sisu-manual +spine -v --html --html-link-search ${SpinePOD}/* if configuration has not been set or to overide the set configuration specify - the output path as well as @@ -217,28 +220,34 @@ note: ~webDocRoot should be the path to web doc root, provide a suitable output *** create db -if there is no sqlite db you first need to create one, to do so +If there is no sqlite db, you first need to create one (an empty db - tables & +indexes), to do so you must specify: + +- the spine binary (executable) - the name of the db and -- the root path for document output -must be specified: +- the path for where the db is to be built - spine -v \ - --sqlite-db-create --sqlite-db-filename="spine.search.db" \ - --output=/var/www/html \ - ~spineMarkupSamples/pod/* +(& you must of course have write permission): - spine -v --sqlite-db-create --sqlite-db-filename="spine.search.db" --output=`echo ~webDocRoot` +spine -v --sqlite-db-create --sqlite-db-filename="spineishearch.db" --sqlite-db-path="/var/www/sqlite" -if you have a configration file providing this information that is to be used -for a document collection you can point to the document collection: +If you have a configration file providing this information that is to be used +for a document collection you can point to the document collection (where the +configuraton file "config_local_site" will be looked for in the .dr +sub-directory): + +spine -v --sqlite-db-create ${SpinePOD} - spine -v --sqlite-db-create ~spineMarkupSamples/pod +To drop (destroy) and re-create a db, you instead would use: --sqlite-db-recreate *** populate db -must specify: -- the name of the db and -- the root path for document output +To populate a db with documents prepared for sisudoc-spine, you must specify: +- the spine binary (executable) +- the name of the db +- the path to the db +- the (recognized) path to a prepared (spine marked up) document or document +- and the root path for document output spine -v --sqlite-update \ --sqlite-db-filename="spine.search.db" \ @@ -252,25 +261,27 @@ for a document collection you can point to the document collection: spine -v --sqlite-update ~spineMarkupSamples/pod/* + ${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionRemote}" --curate --sqlite-update --sqlite-kb-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/* + *** generate a cgi search form in d - spine -v --cgi-search-form-codegen \ - --output=/var/www/html \ - ~spineMarkupSamples/pod - - spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod - - spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site - - spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod - - spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot` - - spine -v --cgi-search-form-codegen \ - --sqlite-db-filename="spine.search.db" \ - --cgi-sqlite-search-filename="spine-search" \ - --output=/var/www/html \ - ~spineMarkupSamples/pod +spine -v --cgi-search-form-codegen \ + --output=/var/www/html \ + ~spineMarkupSamples/pod + +spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod + +spine -v --cgi-search-form-codegen --config=~spineMarkupSamples/pod/.dr/config_local_site + +spine --cgi-search-form-codegen --output=`echo ~webDocRoot` ~spineMarkupSamples/pod + +spine --cgi-search-form-codegen --cgi-sqlite-search-filename="spine_search" --output=`echo ~webDocRoot` + +spine -v --cgi-search-form-codegen \ + --sqlite-db-filename="spine.search.db" \ + --cgi-sqlite-search-filename="spine-search" \ + --output=/var/www/html \ + ~spineMarkupSamples/pod **** compile the cgi search form @@ -302,123 +313,177 @@ cgi-bin directory *** create db & search form - spine -v \ - --sqlite-db-create --sqlite-db-filename="spine.search.db" \ - --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \ - --output=/var/www/html \ - ~spineMarkupSamples/pod/* +spine -v \ + --sqlite-db-create --sqlite-db-filename="spine.search.db" \ + --cgi-search-form-codegen --cgi-sqlite-search-filename="spine-search" \ + --output=/var/www/html \ + ~spineMarkupSamples/pod/* + +*** html with links to search form + +${SpineBIN} -v --epub --html --html-link-curate --curate --output=${SpineOUT} ${SpinePOD}/* + +${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-pdf --html-link-curate --html-link-markup --curate --output=${SpineOUT} ${SpinePOD}/* + +${SpineBIN} -v --source --pod --latex --latex-init --epub --html --html-link-search --html-link-pdf --html-link-curate --html-link-markup --cgi-sqlite-search-filename="${SpineCGIform}" --cgi-url-action="${SpineSearchActionLocal}" --curate --sqlite-update --sqlite-db-filename="${SpineSQLdb}" --output=${SpineOUT} ${SpinePOD}/* + +spine -v --html \ + --html-link-search \ + --output=`echo ~webDocRoot` \ + ${SpinePOD}/* -* Commands +** commands help for a list of commands from the program type: spine -h -at the time of writing this provides the following output: - --abstraction document abstraction - --allow-downloads allow downloads (includes cgi.d from github) - --assert set optional assertions on - --cgi-bin-root path to cgi-bin directory - --cgi-url-root url to cgi-bin (to find cgi-bin) - --cgi-url-action url to post to cgi-bin search form - --cgi-search-title if generating a cgi search form the title to use for it - --cgi-sqlite-search-filename =[filename] default is spine-search - --concordance file for document - --curate extract info on authors & topics from document header metadata - --curate-authors extract info on authors from document header metadata - --curate-topics extract info on topics from document header metadata - --dark alternative dark theme - --digest hash digest for each object - --epub process epub output - --generated-by generated by headers (software version & time) - --hide-ocn object cite numbers - --html process html output - --html-link-curate place links back to curate in segmented html - --html-link-markup provide html link to markup source, shared optionally - --html-link-pdf provide html link to pdf a4 & letter output - --html-link-pdf-a4 provide html link to pdf a4 output - --html-link-pdf-letter provide html link to pdf letter size output - --html-link-search html embedded search submission - --html-seg process html output - --html-scroll process html output - --lang =[lang code e.g. =en or =en,es] - --latex latex output (for pdfs) - --latex-color-links mono or color links for pdfs - --latex-init initialise latex shared files (see latex-header-sty) - --latex-header-sty latex document header sty files - --light default light theme - --manifest process manifest output - --ocn-off object cite numbers - --odf open document format text (--odt) - --odt open document format text - --output =/path/to/output/dir specify where to place output - --parallel parallelisation - --parallel-subprocesses nested parallelisation - --pdf latex output for pdfs - --pdf-color-links mono or color links for pdfs - --pdf-init initialise latex shared files (see latex-header-sty) - --pod spine (doc reform) pod source content bundled --q --quiet output to terminal - --section-backmatter document backmatter (default) - --section-biblio document biblio (default) - --section-blurb document blurb (default) - --section-body document body (default) - --section-bookindex document bookindex (default) - --section-endnotes document endnotes (default) - --section-glossary document glossary (default) - --section-toc table of contents (default) - --serial serial processing - --skip-output skip output - --show-config show config - --show-curate show curate - --show-curate-authors show curate authors - --show-curate-topics show curate topics - --show-epub show epub - --show-html show html - --show-latex show latex - --show-make show make - --show-manifest show manifest - --show-metadata show metadata - --show-pod show pod - --show-sqlite show sqlite - --show-summary show summary - --source document markup source - --set-digest default hash digest type (e.g. sha256) - --set-papersize default papersize (latex pdf eg. a4 or a5 or b4 or letter) - --set-textwrap default textwrap (e.g. 80 (characters) - --sqlite-discrete process discrete sqlite output - --sqlite-db-create create db, create tables - --sqlite-db-drop drop tables & db - --sqlite-db-filename sqlite db to create, populate & make available for search - --sqlite-db-path sqlite db path - --sqlite-db-recreate create db, create tables - --sqlite-delete sqlite output - --sqlite-insert sqlite output - --sqlite-update sqlite output - --www-http http or https - --www-host web server host (domain) name - --www-host-doc-root web host host (domain) name with path to doc root - --www-url-doc-root e.g. http://localhost - --text text output - --theme-dark alternative dark theme - --theme-light default light theme - --txt text output --v --verbose output to terminal - --very-verbose output to terminal - --workon (reserved for some matters under development & testing) - --xhtml xhtml output - --config =/path/to/config/file/including/filename - --debug debug - --debug-curate debug curate - --debug-curate-authors debug curate authors - --debug-curate-topics debug curate topics - --debug-epub debug epub - --debug-harvest debug harvest - --debug-html debug html - --debug-latex debug latex - --debug-manifest debug manifest - --debug-metadata debug metadata - --debug-pod debug pod - --debug-sqlite debug sqlite - --debug-stages debug stages --h --help This help information. +* Configure + +* command line instruction + +Many configuration options can be passed directly from the command line using +command line flags. Evident from the examples given of basic commands. + +* configure environment + +These examples assume the file layout suggested in cloning the git.sisudoc.org +repository, i.e. that the directories sisudoc-spine and sisudoc-spine-samples +are next to each other on a directory tree. Assuming this to be the case, you +may wish to set the following exports with adjustments accoring to your specific +needs for these examples. + +# ❯❯ set spine binary location: +export SpineBIN=./result/bin/spine + +# ❯❯❯ nix builds spine binary: +#export SpineBIN=./result/bin/spine +# ❯❯❯ dub builds spine binary (name depends on build, check): +#export SpineBIN=./bin/spine +#export SpineBIN=./bin/spine-ldc +#export SpineBIN=./bin/spine-dmd + +# ❯❯ location of source files: +export SpineDOC=../sisudoc-spine-samples + +# ❯❯ location of source files pod: +export SpinePOD=${SpineDOC}/markup/pod + +# ❯❯ sisudoc-spine output processing path: +export SpineOUT=./OUTPUT_TEST_sisudocSpine +# ❯❯ sisudoc-spine output processing path (web server e.g.): +#export SpineOUT=/srv/www/spine + +# ❯❯ url to activate search (as configured on web server) +export SpineSearchActionLocal='http://localhost/spine_search' +export SpineSearchActionRemote='https://sisudoc.org/spine_search' + +# ❯❯ path configured for cgi search form: +export SpineCGIform='spine_search' + +# ❯❯ search form db name: +export SpineSQLdb='spine.search.db' + +# ❯❯ configuration cgi search form path: +#export SpineCGIbin=/var/www/cgi/cgi-bin + +# ❯❯ configuration db path: +#export SpineDBpath=/var/www/sqlite + +* configuration files + +Configuration files are yaml files + +The following paths are searched: + + ~/.dr/config_local_site + ~/path_to_pod_root/.dr/config_local_site + +e.g. processing + + ~spineMarkupSamples/pod/* + +will search: + + ~spineMarkupSamples/pod/.dr/config_local_site + + ~/.dr/config_local_site + +to specify an alternative configuration file to use on the command line (in this +example named "my_config"): + + spine -v --html --config=~spineMarkup/pod/.dr/my_config + +here are two sample configuration files + +sample 1. a localhost (check your paths): + +flag: + act0: "--html" + act1: "--html --epub" +output: + path: "/var/www/html" +default: + language: "en" + papersize: "a4" + text_wrap: "80" + digest: "sha256" +webserv: + http: "http" + host: "localhost" + data_http: "http" + data_host: "localhost" + data_root_url: "http://localhost" + data_root_path: "/var/www/html" + data_root_part: "" + images_root_part: "image" + cgi_search_form_title: "≅ SiSU Spine search ፨" + cgi_http: "http" + cgi_host: "localhost" + cgi_bin_url: "http://localhost/cgi-bin" + cgi_bin_subpath: "/cgi-bin" + cgi_bin_path: "/usr/lib/cgi-bin" + cgi_search_script: "spine-search" + cgi_search_script_raw_fn_d: "spine_search.d" + cgi_port: "" + cgi_user: "" + cgi_action: "http://localhost/cgi-bin/spine-search" + db_sqlite: "spine.search.db" + db_pg_table: "" + db_pg_user: "" + +sample 2. sisudoc: + +flag: + act0: "--html" + act1: "--html --epub" +output: + path: "/srv/www/spine" +default: + language: "en" + papersize: "a4,letter.portrait,b4.portrait" + text_wrap: "80" + digest: "sha256" +webserv: + http: "https" + domain: "sisudoc.org" + data_http: "https" + data_domain: "sisudoc.org" + data_root_url: "https://sisudoc.org" + data_root_path: "/srv/www/spine" + data_root_part: "/spine" + images_root_part: "image" + cgi_search_form_title: "≅ SiSU Spine search" + cgi_http: "https" + cgi_domain: "sisudoc.org" + cgi_bin_part: "cgi-bin" + cgi_bin_path: "/var/www/cgi/cgi-bin" + cgi_search_script: "spine_search" + cgi_search_script_raw_fn_d: "spine_search.d" + cgi_port: "" + cgi_user: "" + cgi_action: "https://sisudoc.org/spine_search" + db_sqlite_filename: "spine.search.db" + db_sqlite_path: "/var/www/sqlite" + db_pg_table: "" + db_pg_user: "" |