NAME
Cavil::Matcher - Next-generation license pattern matcher for Cavil
SYNOPSIS
use Cavil::Matcher;
my $m = Cavil::Matcher::init_matcher;
$m->add_pattern(1, Cavil::Matcher::parse_tokens('Permission is hereby granted $SKIP30 to deal'));
my $matches = $m->find_matches('source/file.c'); # [[pattern_id, start_line, end_line], ...]DESCRIPTION
Cavil::Matcher turns source files into license and keyword matches for Cavil. It keeps the proven token-hash prefix-tree algorithm of its predecessor, Spooky::Patterns::XS, but stores the compiled patterns as immutable, memory-mapped segments described by a small manifest, so that adding or removing a pattern never rebuilds the whole cache and index workers share one physical copy of the data per host.
The tokenizer and hashing are a frozen C++ core, bit-for-bit compatible with the previous engine; the segment lifecycle is pure Perl (see Cavil::Matcher::Index and Cavil::Matcher::Manifest). For the design and rationale see docs/Architecture.md.
FUNCTIONS
init_matcher
my $matcher = Cavil::Matcher::init_matcher;Create a new matching engine (a "Cavil::Matcher::Engine").
parse_tokens
my $tokens = Cavil::Matcher::parse_tokens($pattern_text);Tokenize pattern text into the arrayref of token hashes that "add_pattern" expects. A $SKIP wildcard ($SKIP followed by a number n up to 99) is recognised here: each matches from one up to n arbitrary words (at least one, at most n - never a zero-word gap). A pattern may not begin or end with a skip.
Input must be text. This and "normalize" take the string with C-string semantics, so an embedded NUL byte terminates the input (everything after it on that call is ignored). That is fine for their intended use - curated pattern text and human-readable text - and matches the previous engine. To scan raw source bytes (which may contain NULs), use "add_pattern" + "find_matches", whose file reader is NUL-tolerant and reads past embedded NULs.
normalize
my $rows = Cavil::Matcher::normalize($text); # [[line, token, hash], ...]Tokenize text, returning each token with its line number and hash. Text input only - see the NUL-handling note under "parse_tokens".
distance
my $edits = Cavil::Matcher::distance($norm_a, $norm_b);An approximate token-level edit distance between two "normalize" results. Note: for byte-for-byte parity it deliberately reproduces the previous engine (Spooky::Patterns::XS), including its off-by-one (it compares count-1 tokens), so it is not a strict Levenshtein distance and can report 0 for inputs that differ only in a single or trailing token. It is currently unused within Cavil; once the previous engine is retired it can be made a strict Levenshtein.
read_lines
my $rows = Cavil::Matcher::read_lines($file, \%wanted_lines); # [[line, value, text], ...]Return the requested lines of a file as raw (undecoded) bytes. Note: %wanted_lines is consumed - each line found is deleted from the hash (an early-exit optimization), so pass a fresh hash if you need to reuse it.
The returned text for a single physical line is capped at 1 MiB, to bound memory on pathological input (e.g. a minified or binary file that is one enormous line). "find_matches" numbers such a line the same way, so a match reported past the first 1 MiB of a single line can point at text this call does not return. Real, line-wrapped source is unaffected; this only matters for degenerate multi-megabyte lines.
init_hash
my $hash = Cavil::Matcher::init_hash($seed1, $seed2);Create a streaming 128-bit content hash (a "Cavil::Matcher::Hash").
init_bag_of_patterns
my $bag = Cavil::Matcher::init_bag_of_patterns;Create a tf-idf "closest match" model (a "Cavil::Matcher::Bag").
Cavil::Matcher::Engine
The matching engine returned by "init_matcher".
- add_pattern
-
$engine->add_pattern($id, $tokens);Add a pattern (from "parse_tokens") to the in-memory delta segment.
- find_matches
-
my $matches = $engine->find_matches($file);Scan a file and return the resolved matches as
[[pattern_id, start_line, end_line], ...]. - dump($file) / load($file)
-
Write the in-memory patterns to a compiled segment file / replace the engine's state with a single mmapped segment file.
- attach($file)
-
Memory-map an additional compiled segment into the active set. Returns false (without dying) on a missing or invalid file.
- set_tombstones(\@pattern_ids)
-
Drop these pattern ids from results before overlap resolution.
- set_generation($n) / generation
-
Record, and read back, the manifest generation this engine was built from. Reading it back from the engine itself (rather than re-reading the index) is race-free, so a report can record exactly the generation it scanned with even if the index is updated concurrently.
Cavil::Matcher::Hash
Streaming content hash from "init_hash": add($bytes), hash128 ([hi, lo]), hex (32-char string) and hash64.
Cavil::Matcher::Bag
tf-idf closest-match model from "init_bag_of_patterns": set_patterns(\%id_to_text), best_for($text, $count), dump($file) and load($file). dump and load return true on success; a failed load (missing/truncated file) leaves the model unchanged.
SEE ALSO
Cavil::Matcher::Index, Cavil::Matcher::Manifest, https://github.com/openSUSE/cavil.
COPYRIGHT AND LICENSE
Copyright (C) SUSE LLC. This is free software, licensed under GPL-2.0-or-later.