/ 
oEdtk-2.1126
/ README.md

oEdtk

oEdtk is a Perl 5 toolkit for industrial printing processing. It provides the generic building blocks needed to write applications that parse fixed-record data streams (COBOL copybooks, XML or plain text) and turn them into composed documents (LaTeX/PDF, CSV, Excel) and database content: letters, mailings, invoices, banking statements, and similar high-volume productions.

Main features at a glance:

Table of contents

Presentation

oEdtk is not a single program: it is a library plus a set of shipped helper scripts. The core (lib/oEdtk.pm and the modules under lib/oEdtk/) is generic and self-contained; an application is a small Perl script you write that combines the pieces. The main entry point of an application is oEdtk::Main, which reads the configuration, opens the input/output handles and drives the record loop and the composition back-end.

A companion distribution, oEUser, carries the modules that only specific applications need (user-side address helpers, generated record descriptors, editor-specific XML readers, and the document-injection client). It is a separate distribution and is not a prerequisite of oEdtk: the thin oEdtk::* delegates resolve the corresponding oEUser::* modules on demand, and fail with an explicit message when the companion is absent. See oEUser/README.

This document covers the repository and the shipped Perl package (README.md is included in the distribution tarball).

Quick start

The shortest path to a first run:

  1. Prerequisites - a Perl 5 interpreter (see Requirements) and the XML-LibXML module. On Debian/Ubuntu: sudo apt-get install perl libxml-libxml-perl.

  2. Install - from CPAN:

    cpan oEdtk

    or, from a checkout:

    perl Makefile.PL && make && make test && make install

  3. Verify the module loads and reports its version:

    perl -MoEdtk -e 'print oEdtk_release()'

    Expected output: a banner line (c) 2005-2026 oedtk@free.fr - oEdtk v2.1126.

  4. Write a minimal application (myapp.pl) using the fixed-record model (see lib/oEdtk.pm SYNOPSIS):

    use strict;
use oEdtk::Main;

oe_new_job( $ARGV[0], $ARGV[1] );      # input file, output file
oe_rec_motif( "016", "A67 A20 A*" );   # declare record keyed '016'
oe_rec_output( "016", "<SK>%s<#DATE=%s><SK>%s" );

while ( my $line = <IN> ) {
    chomp $line;
    oe_process_ref_rec( 0, 3, $line );  # offset, key length, line
}
oe_compo_link( $ARGV[0], $ARGV[1] );    # close handles, finalise

  5. Run it:

    perl myapp.pl input.txt output.tex

    myapp.pl reads input.txt, unpacks every 016 record, builds the declared output, and writes it to output.tex. Producing a PDF additionally requires a LaTeX toolchain and a populated configuration (see Configuration); without them, the composition back-end cannot run.

Requirements

Operating systems

oEdtk is pure Perl 5 and runs on Linux, macOS and Windows. The shipped helper scripts and the shipped edtk.ini template assume Unix-style paths (/usr/bin/perl, /usr/bin/gs, /usr/bin/pdftk); on Windows they must be adapted. There is no stated minimum OS version.

Runtime

System libraries

Perl dependencies

Declared in Makefile.PL (PREREQ_PM) and mirrored in the generated cpanfile. Key ones, with the pinned floors:

| Module | Floor | Used for | |---|---|---| | Config::IniFiles | 2.30 | reading edtk.ini | | DBI | 1.60 | all database access | | XML::LibXML | 1.70 | XML input | | XML::XPath | 1.13 | XPath queries | | Text::CSV | 1.20 | CSV parsing | | Spreadsheet::WriteExcel | 1.00 | Excel output | | Math::Round | 0.06 | numeric formatting |

Plus, among others: Archive::Zip, Date::Calc, Email::Sender::Simple and Email::Sender::Transport::SMTP, JSON, JSON::PP, Data::UUID, File::Slurp, IPC::Run3, LWP::UserAgent and the HTTP::* modules, IO::Socket::SSL, Log::Log4perl, PDF::Create, PDF::Tk, Readonly, Regexp::Common, Term::ReadKey, Text::CSV_XS, Text::QRCode, Time::HiRes, Try::Tiny, YAML::Tiny, and CGI + CGI::FormBuilder (the CGI front-ends load these at the top level, so they are hard requirements). A full install pulls all of them.

Database drivers

DBI is a hard requirement; the driver depends on the engine:

External programs (per feature)

Development and test extras

Perl::Tidy, Perl::Critic, Test::Fatal, Test::MockModule, Test::Fake::HTTPD, Test::More >= 1.302, IPC::Run3, DBD::mysql, DBD::Pg (declared in TEST_REQUIRES).

Installation

Methods are listed from the simplest to the most advanced.

From CPAN (recommended)

cpan oEdtk        # or: cpanm oEdtk

Pulls the dependencies and installs the modules. Verify: perl -MoEdtk -e 'print oEdtk_release()'.

From source (a checkout of this repository)

perl Makefile.PL
make                # use 'dmake' on Win32
make test
make install
make clean

If libxml2 is in a non-standard location, pass include/library paths:

perl Makefile.PL LIBS='-L/usr/lib' INC='-I/usr/include/libxml2'

Verify: make test reports the test count, and perl -MoEdtk -e1 loads cleanly.

Windows (ActivePerl / PPM)

Install XML-LibXML (and dmake) with PPM, then run the same source steps with dmake instead of make.

Optional companion distribution

oEUser is a separate distribution (see oEUser/README); it requires oEdtk 2.1120 or later. Install it with the same perl Makefile.PL && make && make install sequence from the oEUser/ directory.

Configuration

Where the configuration comes from

The configuration is an INI file (edtk.ini) read by oEdtk::Config::config_read (lib/oEdtk/Config.pm). Resolution order:

  1. edtk.ini next to the running script ($Bin/edtk.ini);
  2. otherwise <libdir>/iniEdtk/edtk.ini;
  3. otherwise the shipped template <libdir>/iniEdtk/tplate.edtk.ini.

The iniEdtk key of the host-name section ([<UPPERCASE-HOSTNAME>]) can redirect the read to another file; the literal value local disables the redirection. Merging is [DEFAULT] -> [ENVDESC] -> the sections the caller requests -> the host-name section (last wins). config_read injects EDTK_HOST, USERNAME, EDTK_PRGNAME (from $0) and EDTK_PRGPATH (the script directory, i.e. $Bin), and expands $name references.

The template

Start from the shipped template lib/oEdtk/iniEdtk/tplate.edtk.ini and adapt it. Its sections:

| Section | Role | |---|---| | [DEFAULT] | Global starting values (corporation, dictionaries, language, log level). | | [ENVDESC] | The runtime directory model (EDTK_DIR_*, file stems, extensions, timeouts). | | [EDTK_DB] | Database DSNs, users, retention, primary-key mode, bulk-load tuning. | | [EDTK_STATS] | Statistics DSNs and canned SQL queries. | | [MAIL] | Mail recipients, sender, subject, SMTP host. | | [COMPO] | LaTeX back-end, commands, line cutoff, batch-id format. | | [EDOCMNGR] | EDMS/GED index columns, formats, FTP host, lookup URL. | | [SERVICES] | Optional network services - inactive by default. | | [MYHOSTNAME] | Example host-name override section (always uppercase). |

Key lexicon

Every key the shipped template sets, by section, with what it drives and the values that change behaviour. A $name value (for example $EDTK_DIR_BASE) is expanded by config_read; an empty Special values cell means the value is free-form or has no special literal.

[DEFAULT]

| Key | Usage | Special values | |---|---|---| | iniEdtk | Redirects the read to another ini file; also read from the host-name section. | local disables the redirect. | | EDTK_CORP | Corporation name printed in the generated documents. | | | EDTK_DICO | Main dictionary (accents, entities) loaded by the text layer. | | | EDTK_DICO_POST | Postal-code dictionary. | | | EDTK_SNIP_ADDR | Address-snipping dictionary. | | | EDTK_SNIP_RATE | Address-snipping rate. | 0 disables snipping. | | EDTK_LANG | Language, i.e. the dictionary section to read. | e.g. EN, FR. | | EDTK_LOG_LVL | Log sensitivity read by oEdtk::logger (its $LOGGERLEVEL). | 4 is the mainline default. |

[ENVDESC] - directories, file stems, extensions and timeouts. Every path is written relative to EDTK_DIR_BASE (default $EDTK_PRGPATH, the running script's directory) and to EDTK_PRGNAME (injected at runtime).

| Key | Usage | Special values | |---|---|---| | EDTK_DIR_BASE | Root of the whole path tree. | default $EDTK_PRGPATH. | | EDTK_DIR_APP, EDTK_DIR_SCRIPT | Application root / per-application script directory. | | | EDTK_DIR_CONFIG | Configuration files and dictionaries. | | | EDTK_DIR_APPTMP, EDTK_DIR_DB | Temporary work area / database files. | | | EDTK_DIR_DATA_IN | Input flux directory. | | | EDTK_DIR_LIB, EDTK_DIR_TMPLATE | Shared libraries / document templates. | | | EDTK_DIR_OUTMNGR, EDTK_DIR_EDOCMNGR, EDTK_DIR_DOCLIB | Output, EDMS transit and document library. | | | EDTK_DIR_LOG, EDTK_DIR_DOCSCRIPT | Logs / per-application document area. | | | EDTK_DOCLIB_LOCK | Document-library lock file. | | | EDTK_FDATAIN, EDTK_FDATAOUT, EDTK_FDATWORK | Input / output / work file stems. | | | EDTK_TYPE_ENV | Environment label, prefixed to mail subjects by the messenger. | Dev (default), Test, Production. | | EDTK_OUTMNGR_MERGE | Whether the outgoing-routeur output is merged. | no (default) / yes. | | EDTK_PRGNAME | Running program name (injected). | | | EDTK_VCS_LOCATION | Version-control location recorded in outputs. | | | EDTK_WATERMARKTEXT | Watermark stamped on generated documents. | | | EDTK_BIN_PERL | Path to the Perl interpreter used when bundling. | | | EDTK_EXT_COMP_OMGR | Extension of the routeur-composed files. | omgr. | | EDTK_EXT_COMPO | Extension of a composed file. | defaults to $EDTK_EXT_LATEX. | | EDTK_EXT_DATA, EDTK_EXT_DEFAULT | Data-file extension / default extension. | DAT / $EDTK_EXT_PDF. | | EDTK_EXT_EXCEL, _LATEX, _PCL, _PDF, _PERL, _WORK | Extensions per artefact type. | xls, tex, pcl, pdf, pl, txt. | | EDTK_WAITRUN | Seconds to wait for a child process. | default 8. | | EDTK_XLS_MAX_ROW | Maximum rows read from an XLS sheet. | default 30000. |

[EDTK_DB] - database DSNs, credentials, retention, primary-key mode and bulk-load tuning.

| Key | Usage | Special values | |---|---|---| | EDTK_DBI_DSN (+ _USER, _PASS) | Active DSN and credentials of the main database. | | | EDTK_DBI_DSN_BAK (+ _USER, _PASS) | DSN and credentials of the backup database. | | | EDTK_DBI_PARAM | DSN of the parameter database. | | | EDTK_DBI_ACQUIT, _DISTRIB, _DISTRIB_STATS, _OUTMNGR, _OUTMNGR_STATS, _TRACKING | Logical table names of the acquittal, distribution, distribution-stats, outgoing, outgoing-stats and tracking tables. | | | EDTK_MAX_USER_KEY | Number of user tracking keys accepted. | 10 (mainline) / 5 (fork). | | EDTK_TRACKING_KEYS | Comma-separated extra tracking columns. | | | EDTK_TRACK_MODE | Tracking back-end. | SQLite (default) / none (tracking off). | | EDTK_TRACK_SIG | Which signals the tracker catches. | warn (all) / halt (die only); no disables. | | EDTK_DB_MAX_DAYS_KEPT, _KEPT_STATS, _TRACKED, _AGREGED | Retention in days of the main, stats, tracking and aggregated tables. | 2, 100, 400, 600. | | EDTK_DB_CHECK_AUTO | Enables the automatic consistency check. | NO (default) / YES. | | EDTK_DB_CHECK_MONTH_DAY | Day of month for the automatic check. | default 15. | | EDTK_DB_PK_IDJOB | Keeps ed_idjob in the INDEX primary key. | yes (mainline) / any other value (fork schema). | | EDTK_DB_IDJOB_DEFAULT | Value written to ed_idjob when the key is kept. | empty until the S51 gate. | | EDTK_DB_FetchHashKeyName | Case-folding of fetched column names (FetchHashKeyName). | NAME_uc (default). | | EDTK_DB_COPY_ENCODING | Source encoding of the CSV bulk-load path. | UTF8 = file already UTF-8; empty = legacy (Latin-1). | | EDTK_DB_DEFER_INDEX_RATIO, _MIN_ROWS | Index-deferral policy of the PostgreSQL bulk load (opt-in). | empty = off; when set, defaults 0.1 / 10000. | | EDTK_DB_BURST_MO | ELT staging threshold, in MiB, on PostgreSQL. | 50 (default); 0 disables; empty/invalid falls back to 50. |

[EDTK_STATS]

| Key | Usage | Special values | |---|---|---| | EDTK_DBI_STATS, EDTK_DSN_STATS (+ _USER, _PASS) | DSN and credentials of the statistics database. | default to the main DSN. | | EDTK_STATS_TRACKING, _OUTMNGR, _DISTRIB, _ACQUIT | Statistics table names. | | | EDTK_STATS_MANUFACT | Manufacturer filter of the outgoing statistics. | comma- or space-separated ED_IDMANUFACT codes; empty = all. | | EDTK_CSV_HEADER_METADATA, EDTK_CSV_HEADER_DISTRIB | Column order of the CSV exports. | | | EDTK_STATS_SQL00..SQL12, SQL90 | Named canned SQL queries of the statistics panel. | SQL90 may name a .sql file ($EDTK_PRGPATH/sql/...). |

[MAIL]

| Key | Usage | Special values | |---|---|---| | EDTK_MAIL_TO, EDTK_MAIL_TST | Production / test recipient. | | | EDTK_MAIL_SENDER | Envelope sender. | | | EDTK_MAIL_SUBJ | Subject template. | %date and %time are substituted. | | EDTK_MAIL_SMTP | SMTP relay host. | | | EDTK_MAIL_REFER, EDTK_MAIL_OMGR | Body templates of the recap / outgoing mail. | | | EDTK_MAIL_LIST_1.._5 | Named distribution lists. | |

[COMPO] - the LaTeX back-end.

| Key | Usage | Special values | |---|---|---| | EDTK_BIN_COMPO | Directory holding the TeX binaries. | | | EDTK_COMPO | The composition command. | e.g. pdflatex -halt-on-error. | | EDTK_COMPO_CMD_PDF, _DVI, _DVIPDF | Per-format composition commands. | | | EDTK_COMPO_LIB | Path to the edtkdoc.cls class file. | | | EDTK_FILIERES_RECTO_VERSO | Channels whose index is padded to an even page count. | empty = none. | | EDTK_CMD_RUN_BG_NOWAIT | Non-blocking background composition. | empty = blocking; set = non-blocking. | | EDTK_LINE_CUTOFF | Maximum characters before a newline in generated documents. | empty falls back to 120. | | EDTK_RP_KILL_NON_LATIN | Strips non-Latin-1 (DEL and C1) characters from input records. | off by default. | | EDTK_IDLDOC_FORMAT | Document batch-id format. | UUID (default) / LEGACY or old (16-digit id, case-insensitive). | | EDTK_COMPO_ASYNC | Asynchronous composition (commented out in the template). | yes / no. |

[EDOCMNGR] - EDMS/GED indexing and transfer.

| Key | Usage | Special values | |---|---|---| | EDMS_INDEX_COLS | Columns of the EDMS index. | | | EDMS_DATE_FORMAT | Date format used in the index. | %Y/%m/%d. | | EDMS_IDX_FORMAT | Index format. | DOCUBASE (default) / SCOPMASTER. | | EDMS_IDX_HEADER | Index header line (;-separated). | | | EDMS_ACTIVATE_HEADER_ALFRESCO | Emits the Alfresco header line. | YES / NO. | | EDMS_FTP_HOST, _PORT, _USER, _PASS | FTP(S) transfer target. | | | EDMS_FTP_DIR_IDX, _DIR_DOCS | Remote index / document directories. | | | EDTK_EDOCS_LINK | Base URL of the document link. | | | EDMS_HTML_HOST | HTML host used to build the lookup URL. | | | EDMS_URL_LOOKUP | Lookup URL template. | %s and %d placeholders. | | EDMS_BIN_GS, EDMS_BIN_PDFTK | Ghostscript / pdftk binaries. | |

[SERVICES] - optional network services. All are inactive by default: an empty value keeps the mainline behaviour.

| Key | Usage | Special values | |---|---|---| | NWS_EMAILING_URI | E-mailing service base URL. | empty = unused (/emails is appended). | | EMAILING_TEMPLATE, EMAILING_USER_AGENT | E-mailing template name / User-Agent string. | empty = each client's shipped default. | | KEYCLOAK_URL, _CLIENT_ID, _SECRET_ID | Keycloak token endpoint and client credentials. | all three required together. | | KEYCLOAK_REALM | Keycloak realm path element. | empty = the client's shipped default. | | DISSEMINATION_PLAN_URL | Dissemination-plan service URL. | empty = the step is skipped. | | EDTK_SSL_VERIFY | TLS peer and host-name verification. | 1 (default, safe) / 0 (private CA only). | | EDTK_SSL_CA_PATH | CA bundle, a file or a directory. | empty = system trust store. | | EDTK_ACQUIT_API | Named acquittal callable, Module::function. | empty = no API acquittal. | | EDTK_MAIL_ACQ | Recipients of the acquittal recap report. | empty = no report. | | APIECM_URL, APIECM_APIKEY_EDITIQUE | ECM API base URL and the static editique API key. | empty = the ECM injector is unusable. |

[MYHOSTNAME] - a worked example of a host-name override section. Section names are always uppercase; a section matching uc(hostname()) is merged last, so it wins, and its iniEdtk key can redirect the whole read.

A few keys the code reads are not shipped in the template and stay optional: EDTK_DICO_COUNTRY (country dictionary), EDTK_DICO_XLAT (LaTeX character translation), EDTK_XML_CATALOG_FILES (XML catalogs, read from [MAIL]), EDTK_DB_STORAGE_ENGINE (MySQL storage engine, default MyISAM), EDTK_ENTIRE_YEARS_KEPT and EDTK_STATS_DAYS_FROM (retention / history windows).

The runtime directory model

[ENVDESC] defines every path relative to EDTK_DIR_BASE (which is set to the script directory at runtime) and to EDTK_PRGNAME (the running program name). The main ones:

Environment variables read by the code

Besides the keys above (which are exported into the process environment at runtime), the code reads a set of $ENV{...} variables, including:

Minimal example (no real secret)

[DEFAULT]
EDTK_CORP      = My Corp
EDTK_LANG      = EN
EDTK_LOG_LVL   = 4

[ENVDESC]
EDTK_DIR_BASE  = $EDTK_PRGPATH
EDTK_DIR_SCRIPT= $EDTK_DIR_BASE/app/$EDTK_PRGNAME

[EDTK_DB]
EDTK_DBI_DSN      = DBI:SQLite:dbname=$EDTK_DIR_SCRIPT/oEdtk_db.tmp
EDTK_DBI_DSN_USER = oedtk
EDTK_DBI_DSN_PASS = change-me

Security

edtk.ini can carry database and service passwords. Keep it out of version control, restrict its permissions, and never commit real credentials. The shipped template uses placeholder values only.

Usage

The application model

An oEdtk application is a small script with a fixed shape (see the Quick start and lib/oEdtk.pm SYNOPSIS):

  1. oe_new_job($infile, $outfile) - read the configuration, open the IN/OUT handles, start the composition back-end.
  2. Declare each record once: oe_rec_motif($key, $template) (mandatory), oe_rec_output($key, $sprintf_template) and, optionally, oe_rec_pre_process / oe_rec_process / recEdtk_post_process to hook user subs.
  3. Read the stream line by line and call oe_process_ref_rec($offset, $keylen, $line) for each - it recognises the record by key, unpacks it into the global @DATATAB, runs the hooks and builds the output.
  4. oe_compo_link($infile, $outfile) - close the handles and finalise (emit the LaTeX trailer, wait for the composer, optionally import the result).

oe_new_job also parses run-time options, among them --index, --massmail, --edms, --cgi, --noinputfiles and --input_code=<code>.

The migration CLI

lib/db_migrate.pl is a standalone command line over oEdtk::DBAdmin that updates a database schema. Exactly one mode is required:

perl lib/db_migrate.pl --check-only --dsn <dsn> [--user U] [--pass P]
perl lib/db_migrate.pl --dry-run    --dsn <dsn>
perl lib/db_migrate.pl --historize  --dsn <dsn> [--stamp YYYYMMDD_HHMMSS]
perl lib/db_migrate.pl --revert     --dsn <dsn> --stamp YYYYMMDD_HHMMSS
perl lib/db_migrate.pl --additive   --dsn <dsn> [--with-constraints]

| Mode | Effect | |---|---| | --check-only | Detect the schema level and print the plan. Nothing is written. | | --dry-run | Like --check-only, plus the copy SQL and a DDL preview. | | --historize | Build a T target, copy by named columns, validate, switch over and keep the old schema as history (recommended). A base already at T is a no-op unless --force. | | --revert | Undo a --historize run (--stamp required). | | --additive | In-place additive update (add column, widening, CREATE ... IF NOT EXISTS) - never a drop, a narrowing or a rename; --with-constraints also changes the key after a duplicate pre-check. |

Options: --dsn, --user, --pass, --stamp, --force, --with-constraints. Exit code 0 on success, 1 on a controlled abort (reason on stderr); --check-only and --dry-run are strictly read-only. The password is never printed.

CGI front-ends

The shipped scripts under lib/oEdtk/lib/cgi-bin/ provide web front-ends: edtk.cgi (inline composition of an uploaded XML), edtk_inged.cgi (push a document into EDMS), edtk_ged.cgi (redirect to the document lookup) and edtk_para.cgi (end-user editor of inline parts). Each is gated by a configuration key named after the script basename, in the [EDOCMNGR] section.

Helper scripts

lib/oEdtk/lib/ ships operational scripts, all run as perl <script> [args] with arguments read positionally or from the configuration: database maintenance (db_*.pl), index/output management (index_*.pl), tracking statistics (tracking_Statistics.pl), acquittal (acq_*.pl), parameters import/export (params_*.pl), and configuration/dev helpers (check_oEdtk_ini.pl, cob2pm.pl, edms_*.pl, ged_process.pl). Shell helpers (*.sh) cover backup and go-live steps.

Inputs and outputs

Use cases

1. Discovery - turn a fixed-record file into a document

Need: convert a small file into a readable document. Steps: install oEdtk, copy a minimal edtk.ini, write a script that declares the record motifs and outputs (see Quick start), run perl myapp.pl input.txt output.tex. Result: output.tex holding one composed block per recognised record. Limits: PDF output needs a LaTeX toolchain and a populated [COMPO] section.

2. Common - composition with output management

Need: produce a batch of documents and register them. Steps: enable --index in oe_new_job, configure [EDTK_DB] / [COMPO], then run the application; oe_compo_link calls oe_after_compo, which can import the result through oEdtk::Outmngr and, when configured, EDMS. Result: composed documents plus index rows in the output-manager database.

3. Advanced - bulk database load

Need: load a large CSV into a table efficiently. Steps: use oEdtk::DBAdmin::csv_import with a PostgreSQL DSN to go through the COPY FROM STDIN path; large files may be routed through the staging "ELT" path (threshold EDTK_DB_BURST_MO, default 50 MiB). Pass force_copy => 1 (--forceCopy in the packaged tracker) to bypass the staging path and force the parsed COPY path regardless of volume. See docs/dbadmin/D80-report.md. Result: the table is loaded; reported row counts match the source.

4. Automation - scripted database maintenance and statistics

Need: version a schema change or report statistics. Steps: run lib/db_migrate.pl --dry-run --dsn <dsn> to preview, then --historize to apply; use index_Statistics.pl / tracking_Statistics.pl for reports. Result: a schema update with a reversible history, or a statistics report on stdout.

Features

Record parsing and output building

Declare a record with oe_rec_motif($key, $template) (an unpack-style template; end it with A* to drop trailing data) and an output with oe_rec_output($key, $sprintf_template). Optional hooks (oe_rec_pre_process, oe_rec_process, recEdtk_post_process) run user subs around unpacking and output; recEdtk_erase / recEdtk_redefine let you ignore or redefine a record mid-stream. Unpacked values live in the global @DATATAB.

LaTeX composition

oEdtk::Run builds and executes the configured LaTeX command (synchronously or through an asynchronous FIFO), and oEdtk::Doc / oEdtk::TexDoc build the LaTeX body (tags, tables, includes, QR codes, escaping).

Output management

oEdtk::Outmngr inserts index lines, assigns lots and filieres, and handles depot, purge and statistics of document batches.

Tracking and statistics

oEdtk::Tracking records document-level events into the tracking database; stats_week, stats_iddest and stats_month produce reports.

Database administration

oEdtk::DBAdmin connects (with a retry loop and a store-and-forward backup buffer), creates and detects schemas, historises/reverts/additively upgrades them, and loads CSV in bulk. PostgreSQL, MySQL/MariaDB, SQLite and Oracle are supported, with per-engine SQL differences (e.g. LOAD DATA INFILE on MySQL, COPY on PostgreSQL).

XML

oEdtk::XML::DTD (parse/validate against a DTD), oEdtk::XML::XSD (against an XSD), oEdtk::XML::Default (Cameleon reader) and oEdtk::XML::Generic (generic reader plus schema verification).

Network services (optional)

oEdtk::Emailing (mailing service), oEdtk::ECM (ECM document injection), oEdtk::Keycloak (token endpoint), oEdtk::DisseminationPlan, and the shared TLS transport oEdtk::Net. All are inactive until configured under [SERVICES].

Extensions

oEdtk::Ext plugs corporate code in without a registry: require_optional loads a companion module by its fixed name (absence is a documented no-op), and load_named_callable / call_named_callable resolve a Module::function named in the configuration after validating its name.

Tests and validation

Layout

Running the suite

tools/run-tnr.sh                       # SQLite/file tests only
tools/run-tnr.sh --db pg               # also against PostgreSQL
tools/run-tnr.sh --db all              # every database engine
tools/run-tnr.sh --no-start --db pg    # assume the database is already up
tools/run-tnr.sh --no-prereqs -- -j4   # skip the pre-flight, pass prove options

tools/run-tnr.sh runs the pre-flight dependency check, optionally starts the Docker test databases, then runs the suites. Under the hood it is:

prove -Ilib -It/lib -r t
prove -Ilib -IoEUser/lib -r oEUser/t      # when oEUser/ is present

Caveat: prove t without -r does not descend into t/int/; always use -r (or the runner). To run one test or category, pass a path, e.g. prove -Ilib -It/lib t/int/15_config.t.

Test databases

Docker-based, one-shot test databases live under tools/db/; they mirror the tracker/dev/build/docker-compose-*.yml set (same container names and ports, so the two never drift):

| Engine | Compose file | Container | Host port | Driver | |---|---|---|---|---| | PostgreSQL | docker-compose-pg.yml | postgres_ephemere | 5432 | DBD::Pg | | MariaDB | docker-compose-mariadb.yml | mariadb_ephemere | 3307 | DBD::mysql | | MySQL | docker-compose-mysql.yml | mysql_ephemere | 3308 | DBD::mysql | | Oracle | docker-compose-oracle.yml | oracle_ephemere | 1521 | DBD::Oracle (optional) |

tools/db/start-db.sh pg        # start (or reuse) the PostgreSQL instance
tools/db/start-db.sh oracle    # first Oracle start creates the PDB (1-3 min)
tools/db/stop-db.sh pg         # stop and remove the volume

start-db.sh is idempotent (an already-running container is reused). Only loopback (127.0.0.1) is exposed. All engines read the same three keys from the git-ignored .env at the repository root (POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD), copied from .env.example; the password is never committed or printed. Prerequisites: Docker (daemon reachable by the working user) and docker compose; images are pulled on first use.

Optional and gated tests

Some tests only run when the matching environment is provided:

Code-quality gates

tools/tidy_all.sh [--dry-run] <path>...   # mechanical perltidy (profile .perltidyrc)
prove -Ilib -It/lib t/int/09_style.t      # perltidy + Perl::Critic (severity 5)
prove -Ilib -It/lib t/int/05_encoding.t   # UTF-8/ASCII + LF everywhere

Interpreting results

A green run reports Files=..., Tests=..., PASS. Ratchet tests fail in both directions - when a guarded item appears, and when a recorded item is already fixed - so a failure means the frozen list and the tree disagree; fix one or the other.

Architecture

Components

Execution flow

flowchart TD
    App["Application script (.pl)"] --> Main["oEdtk::Main - oe_new_job"]
    Config["oEdtk::Config - edtk.ini"] --> Main
    Main --> Parser["Record loop - oe_process_ref_rec"]
    Parser --> Run["oEdtk::Run - oe_compo_run (LaTeX)"]
    Main --> Link["oe_compo_link"]
    Run --> After["oe_after_compo"]
    After --> Omgr["oEdtk::Outmngr"]
    After --> Edms["EDMS / EDOCMNGR"]
    Parser --> Track["oEdtk::Tracking"]
    Omgr --> DB["oEdtk::DBAdmin"]
    Track --> DB

Entry points and extension

The per-application entry point is the application .pl (a template ships as lib/oEdtk/lib/template/runEdtk.pl). The library entry point is oEdtk::Main. The standalone CLI entry point is lib/db_migrate.pl. Corporate extensions attach through oEdtk::Ext and the configuration, not through a compile-time registry.

Repository layout

Makefile.PL  MANIFEST  MANIFEST.SKIP  README.md  Changes   # packaging
lib/oEdtk.pm                                               # main module
lib/oEdtk/                                                 # core modules
lib/oEdtk/iniEdtk/                                         # config templates, dictionaries
lib/oEdtk/lang/                                            # message catalogues (en, es, fr)
lib/oEdtk/lib/                                             # shipped helper scripts + CGIs + templates
lib/db_migrate.pl                                          # schema-migration CLI
t/                                                         # test suite
oEUser/                                                    # optional companion distribution
docs/                                                      # working documentation (not shipped)
tools/                                                     # maintainer tooling (not shipped)

Modules

Grouped by role (path, responsibility, and the main internal dependencies). This is a curated map, not an exhaustive listing.

Core

| Module | Responsibility | Uses | |---|---|---| | lib/oEdtk.pm | Documentation-only top module; oEdtk_release. | - | | lib/oEdtk/Main.pm | Job setup, record loop, output tags, corporation/ID, date helpers. | Config, Run, DisseminationPlan | | lib/oEdtk/Config.pm | Reads edtk.ini (config_read, get_ini_path). | Config::IniFiles, Sys::Hostname | | lib/oEdtk/DBAdmin.pm | DB connect, schema create/detect/upgrade/historise, bulk CSV load, backup buffer. | DBI, Config | | lib/oEdtk/Outmngr.pm | Output-manager index, lots, filieres, depot/purge/stats. | DBAdmin | | lib/oEdtk/Tracking.pm | Document-level tracking and statistics. | Main, DBAdmin, Dict | | lib/oEdtk/Run.pm | Composition runner and post-composition import. | Config, Outmngr, EDMS |

Parsing and document primitives

| Module | Responsibility | |---|---| | lib/oEdtk/Record.pm | An in-memory record: an ordered field list. | | lib/oEdtk/RecordParser.pm | Streaming record parser (motifs, denormalisation). | | lib/oEdtk/Doc.pm | LaTeX document builder (tags, tables, includes, escaping). | | lib/oEdtk/TexDoc.pm | Doc plus QR codes / hhline / TexTag handling. | | lib/oEdtk/LaTeX.pm | Latin-9 escaping and small TeX constructors. |

Data and validation helpers

| Module | Responsibility | |---|---| | lib/oEdtk/Restitution.pm | Read a value out of nested data by key path. | | lib/oEdtk/IBAN.pm | Validate IBAN/BIC. | | lib/oEdtk/FixIdx.pm | Pad each document to an even page count. | | lib/oEdtk/Utils.pm | Text cleanup, index format, date/price parsing, tracking helpers (corporate part delegates to oEUser::Utils). | | lib/oEdtk/Ext.pm | Load and call optional corporate code by name. |

XML

| Module | Responsibility | |---|---| | lib/oEdtk/XML/DTD.pm | Parse and validate XML against a DTD. | | lib/oEdtk/XML/XSD.pm | Validate XML against an XSD. | | lib/oEdtk/XML/Default.pm | Default (Cameleon) XML reader. | | lib/oEdtk/XML/Generic.pm | Generic XML reader plus schema verification. | | lib/oEdtk/XML/DTD/Utils.pm | Delegate to oEdtk::XML::DTD. |

Network and services

| Module | Responsibility | |---|---| | lib/oEdtk/Net.pm | Shared HTTP/TLS transport settings. | | lib/oEdtk/ECM.pm | ECM document injection and mailed-date stamping. | | lib/oEdtk/Emailing.pm | Delegate to oEUser::Emailing. | | lib/oEdtk/Keycloak.pm | Delegate to oEUser::Keycloak. | | lib/oEdtk/DisseminationPlan.pm | Delegate to oEUser::DisseminationPlan. |

Other shipped modules

AddrField, DateField, Dict, EDMS, FatalsToEmail, Field, FPField, Messenger, OmgrIdxDoc, SchemaScan, SignedField, Spool, TexTag, Util, XPath, dataEdtk, libDev, libXls, logger, trackEdtk, tuiEdtk. Two modules are legacy: Tracking2 (superseded by Tracking) and logger2 (a deprecated dead duplicate - use logger).

Public interfaces

The functions an application or an operator is most likely to use. Signatures follow the shipped POD.

| Function | Module | Signature / purpose | |---|---|---| | oe_new_job | oEdtk::Main | oe_new_job($infile, $outfile [, $single_job_id]) - open IN/OUT, start composition. | | oe_process_ref_rec | oEdtk::Main | oe_process_ref_rec($offset, $key_len, $line [, $rec_offset, $rec_len]) - recognise and process one record; returns 1/0. | | oe_compo_link | oEdtk::Main | oe_compo_link($infile, $outfile) - close handles, finalise. | | oe_rec_motif | oEdtk::Main | oe_rec_motif($key, $template) - declare a record's unpack template. | | oe_rec_output | oEdtk::Main | oe_rec_output($key, $sprintf_template) - declare a record's output. | | oe_rec_process / recEdtk_post_process | oEdtk::Main | ( $key, \&sub ) - hook a sub after unpacking / after output. | | config_read / get_ini_path | oEdtk::Config | config_read(@sections, $opt) -> %cfg; get_ini_path() -> resolved ini path. | | db_connect | oEdtk::DBAdmin | db_connect($cfg, $dsn_key) -> $dbh (with fallback to the backup buffer). | | csv_import | oEdtk::DBAdmin | Bulk-load a CSV into a table (per-engine fast path). | | schema_MigrationPlan / schema_Historize / schema_Revert / schema_Additive | oEdtk::DBAdmin | Plan, apply, revert or additively update a schema (drives db_migrate.pl). | | oe_compo_run / oe_cmd_run | oEdtk::Run | Build and run the LaTeX command (sync/async). | | Tracking->new / ->track / ->end | oEdtk::Tracking | Open a tracking session, record an event, close it. | | omgr_* | oEdtk::Outmngr | Index insert, lot/filiere assignment, depot, purge and statistics. | | logger | oEdtk::logger | logger($level, $msg) - emit a diagnostic on stderr. |

Limits and constraints

Troubleshooting

| Symptom | Possible cause | Remedy | |---|---|---| | Can't locate oEdtk/... in @INC | Modules not installed or lib/ not on @INC. | perl -Ilib ... from a checkout, or install the distribution. | | Configuration not found / empty values | No edtk.ini next to the script and none under iniEdtk/. | Copy tplate.edtk.ini to edtk.ini, or fix the iniEdtk redirection. | | XML input fails at startup | XML-LibXML / libxml2 missing or not found. | Install XML-LibXML (PPM/package); rebuild with LIBS/INC if libxml2 is elsewhere. | | Database connect fails | Missing driver or wrong DSN. | Install the matching DBD::*; check the EDTK_DBI_* keys. | | PostgreSQL prints relation ... already exists, skipping | A CREATE ... IF NOT EXISTS on an existing object. | Expected; the shipped DBAdmin pre-checks and logs it at Information level. | | prove t runs only a few files | prove does not recurse without -r. | Use prove -Ilib -It/lib -r t or tools/run-tnr.sh. | | A ratchet test fails | The frozen list and the tree disagree. | Update the tree, or the corresponding t/data/*.txt baseline, in the intended direction. |

Development and contribution

Environment

Clone the repository and install the test extras listed in Requirements. Nothing else is needed to run the suite over SQLite; the other engines come from the Docker test databases (see Tests and validation).

Code conventions

Before contributing

tools/run-tnr.sh                 # full suite (SQLite)
tools/run-tnr.sh --db pg         # plus PostgreSQL
tools/tidy_all.sh lib t          # reformat
prove -Ilib -It/lib t/int/09_style.t t/int/05_encoding.t

Never weaken a test or a ratchet baseline.

Branching

Integration work happens on the integration/2.1120 branch; post-release steps do not bump the version -- a release does, and every module of the package then carries the package version (currently 2.1126). The progress ledger is docs/integration/STATE.md.

API snapshots

tools/api_snapshot.pl produces reproducible JSON inventories of a module tree's public API (exports, versions, load status), used to compare trees. The frozen snapshots live under snap/ and are regenerated from the repository root:

perl tools/api_snapshot.pl lib --out snap/M.json
perl tools/api_snapshot.pl oEdtk-fork-M --extra-inc oEUser-M/lib \
     --home-ini app-test-fusion/edtk.ini --out snap/F.json
perl tools/api_snapshot.pl oEUser-M/lib --extra-inc oEdtk-fork-M \
     --home-ini app-test-fusion/edtk.ini --exclude '::Descriptor::' --out snap/oEUser-M.json

The --extra-inc and --home-ini options compensate for the read-only sources' dependencies; --exclude '::Descriptor::' skips the generated descriptor modules. snap/ is not part of the distribution.

License, security and support

License

oEdtk is released under the Artistic License 2.0, a GPL-compatible free-software license. The full text ships as lib/oEdtk/COPYING; see also the LICENCE and COPYRIGHT sections of lib/oEdtk.pm. It is distributed without any warranty.

Security

Support and contact