NAME

JQ::XS - Perl wrapper for libjq

SYNOPSIS

use JQ::XS;

my $jq = JQ::XS->new('.foo[] | select(. > 2)');

# Perl data interface
my @results = $jq->process({ foo => [1, 3, 5] });
# Returns: (3, 5)

# JSON text interface
my @out = $jq->process_json('{"foo":[1,3,5]}');
# Returns: ('3', '5')

# Named arguments, as jq's --arg/--argjson give you
my $greet = JQ::XS->new('"\($greeting), \(.name)!"',
                        vars => { greeting => 'Hello' });
my ($msg) = $greet->process({ name => 'Alice' });   # "Hello, Alice!"

# Output formatting, as jq's -S/--tab/-r give you
my $pretty = JQ::XS->new('.', tab => 1, sort_keys => 1);
print $pretty->process_json('{"b":1,"a":2}'), "\n";

# Refuse filters that pull in modules from disk
my $safe = do {
    delete local $ENV{HOME};     # ...including the ~/.jq jq adds on its own
    JQ::XS->new($untrusted, allow_includes => 0);
};

# The debug, stderr and input/inputs builtins
my $sum = JQ::XS->new('[., inputs] | add',
                      inputs => [2, 3],
                      debug  => sub { warn "jq: $_[0]\n" });
my ($total) = $sum->process(1);   # 6

# Get the program source
my $prog = $jq->program;

DESCRIPTION

JQ::XS provides a clean object-oriented wrapper around libjq, the C library behind the jq command-line tool. It allows you to:

- Compile and execute jq filter programs - Process Perl data structures (hashes, arrays, numbers, strings) - Process JSON text - Pass named and positional arguments to a filter - Format JSON output the way the jq command line does - Serve the input, inputs, debug and stderr builtins from Perl - Restrict what a filter may load from disk - Handle errors gracefully with Perl exceptions (croak)

METHODS

new($program, %options)

Creates a new JQ::XS object by compiling the given jq filter program.

my $jq = JQ::XS->new('.foo');

Croaks with an error message if the program fails to compile.

The remaining arguments are name/value pairs. Every one of them has a matching method, so anything that can be passed here can also be changed later; the exceptions are vars, args, attrs, library_paths and allow_includes, which take effect while the program is being compiled and so have to be given up front.

vars => \%hash

Named arguments, as jq --arg and jq --argjson pass them. Each key becomes a $variable in the program, and the values are arbitrary Perl data converted the same way process() converts its input.

my $jq = JQ::XS->new('.[] | select(.dept == $dept)',
                     vars => { dept => 'sales' });

args => \@list

Positional arguments, as jq --args passes them. They are available to the program as $ARGS.positional.

$ARGS is always defined, whether or not either option was given: $ARGS.named holds vars and $ARGS.positional holds args, exactly as on the jq command line.

allow_includes => 0

Reject the program if it contains an include or import directive, rather than compiling it and letting it read .jq files from disk. Defaults to true, i.e. modules work as they always have. It does not stop jq importing ~/.jq -- see "RESTRICTING MODULE LOADING" for that.

library_paths => \@dirs

Where include and import look for modules -- jq's -L option, and its JQ_LIBRARY_PATH attribute. See "RESTRICTING MODULE LOADING" for what this does and does not prevent.

Defaults to empty, which is not what the jq command line defaults to: jq adds two directories beside its own binary, and a library embedded in someone else's program has no business reading those uninvited. It does not keep ~/.jq out, which jq imports without consulting the library path at all -- see "RESTRICTING MODULE LOADING".

attrs => \%hash

jq attributes to set before the program is compiled, for anything library_paths does not cover: JQ_ORIGIN, PROGRAM_ORIGIN, and JQ_LIBRARY_PATH itself. See attr() and set_attr().

flags => $flags

Flags for each run of the program, OR'd together from "CONSTANTS". Only jq's execution tracing lives here. See flags().

die_on_halt_error => 1

Turn a filter's halt_error into a Perl exception. See die_on_halt_error().

debug => \&code

stderr => \&code

inputs => \&code | \@values

Perl code behind the debug, stderr and input/inputs builtins, which do nothing at all otherwise. See set_debug_cb(), set_stderr_cb() and set_inputs().

pretty, indent, tab, sort_keys, ascii, color, raw

How process_json() formats its output. See set_output().

process($data)

Processes Perl data through the compiled jq filter. Takes a Perl scalar (which can be a reference to a hash or array) and returns a list of results. Each result is a Perl data structure.

my @results = $jq->process({ name => 'Alice' });

In scalar context, returns an arrayref of results.

my $results_ref = scalar($jq->process($data));

Croaks if the jq filter produces a runtime error or if the processing fails.

Boolean handling

JSON booleans returned by a filter become JSON::PP::Boolean objects, which behave as true/false in boolean context and stringify to 1 and 0. They compare equal to the JSON::PP::true and JSON::PP::false constants.

my ($is_big) = JQ::XS->new('. > 2')->process(5);   # JSON::PP::true

On input, the following are converted to JSON true/false:

JSON::PP::Boolean, Types::Serialiser::Boolean, or boolean objects (by their truth value)
unblessed references to a plain scalar, e.g. \1 and \0
Perl's native boolean values, i.e. the results of comparison and logical operators and of builtin::true/builtin::false
```
$jq->process($x > $y);   # jq sees true or false, not 1 or ""
```
On perls before 5.36 this only works for a boolean passed directly to process(); a copy (e.g. stored in a hash or array first) loses its boolean identity and is treated as an ordinary number/string. On perl 5.36 and later, copies keep their boolean flag and are recognized anywhere in the structure.

process_json($json_text, %options)

Like process, but takes JSON text as input and returns a list of JSON strings (one for each output).

my @json_out = $jq->process_json('{"name":"Alice"}');

Croaks if the JSON input is invalid or if the jq filter produces a runtime error.

Any options given here are merged over the object's own output options for this one call, and are the same ones set_output() takes:

my @pretty = $jq->process_json($json, indent => 4, sort_keys => 1);

program()

Returns the source code of the compiled jq filter program.

my $src = $jq->program;

OUTPUT FORMATTING

set_output(%options)

Sets how process_json() formats its results, and returns the object. Options not mentioned are reset to their defaults, so this replaces the current settings rather than adding to them; to change one and keep the rest, merge "output_options()" yourself, or pass the option to process_json() for a single call.

$jq->set_output(indent => 2, sort_keys => 1);

Each option corresponds to a jq command-line switch:

pretty => 1

Indent output by two spaces, one value per line, as jq does by default. The JQ::XS default is jq's -c: everything on one line.

indent => $n

Indent by $n spaces, $n being 0 to 7 -- jq's --indent n. As in jq, indent => 0 still puts one value per line, just without indentation; it is not the same as compact output.

tab => 1

Indent with tabs -- jq's --tab. Wins over indent and pretty, as it does in jq.

sort_keys => 1

Emit object keys in sorted order -- jq's -S.

ascii => 1

Escape all non-ASCII characters -- jq's -a.

color => 1

Colorize the output with ANSI escapes -- jq's -C. set_colors() chooses the colors.

raw => 1

Emit a result that is a JSON string as the string itself, without quotes or escapes -- jq's -r. Results of every other type are still JSON.

Note that raw with ascii behaves as jq -r -a does, which is to say that ascii wins and strings come out quoted and escaped.

output_options()

Returns a hashref copy of the options set_output() was last given.

HALT AND EXIT

A filter can stop itself with jq's halt or halt_error. Neither is an error as far as process() is concerned -- results produced before the halt are still returned -- so these three methods, all of which describe the most recent run, are how you find out that it happened.

halted()

True if the last run ended in halt or halt_error.

exit_code()

The exit code the filter halted with, or undef if it did not halt (or halted without one). This is what the jq command line would have exited with.

error_message()

The message halt_error was given, or undef. Usually a string, but halt_error accepts any JSON value and this returns it converted like any other result.

die_on_halt_error($bool)

Sets whether halt_error raises a Perl exception carrying its message, and returns the previous setting. Off by default, which is why the message would otherwise be silently dropped -- there is no stderr for it to go to. Plain halt never raises, having no message; use "halted()" for that.

$jq->die_on_halt_error(1);
my @out = eval { $jq->process($data) };
warn "filter gave up: $@" if $@;

CALLBACKS

libjq leaves the debug, stderr, input and inputs builtins unconnected, so until you give them somewhere to go, debug and stderr discard their input and inputs produces nothing.

An exception thrown by a callback is not lost, but it cannot be raised from where it happens either -- that would abandon a jq program mid-run. A debug or stderr exception is re-raised by process() once the run has finished; an inputs exception becomes a jq error, which the filter's own try/ catch can see, and which otherwise surfaces as the run's runtime error.

set_debug_cb(\&code)

Sets the code called for each debug in the filter, with the value being debugged as its only argument. Pass no argument, or undef, to disconnect it again. Returns nothing.

$jq->set_debug_cb(sub { warn 'DEBUG: ' . to_json($_[0]) . "\n" });

set_stderr_cb(\&code)

The same, for the stderr builtin.

jq grew jq_set_stderr_cb in 1.7, so under JQ_SYSTEM=1 against an older libjq this croaks rather than silently doing nothing, and features() reports stderr_cb as false. The vendored build always has it.

set_inputs($source)

Connects the input and inputs builtins to $source, and returns the object. $source is either an arrayref, whose elements are handed out in order, or a code ref called once per value:

$jq->set_inputs(sub { my $line = <$fh>; defined $line ? $line : () });

Returning an empty list ends the stream. Returning undef does not: that is one more value, JSON null. Pass undef as $source to disconnect.

RESTRICTING MODULE LOADING

A jq program can pull definitions in from .jq files on disk:

include "foo";                     # ./foo.jq, or a library path
import "bar" as bar {search:"/x"}; # /x/bar.jq

Which is worth thinking about if the programs you compile come from somewhere you do not control.

A definition gets in from disk two ways -- a directive in the program, or the ~/.jq jq imports on its own -- and they need different answers. Both happen while the program is being compiled, which is to say inside new(): nothing is loaded later, and nothing you change on an object afterwards unloads what it compiled with.

Directives in the program

allow_includes => 0 rejects any program containing one of these directives, at compile time, before jq has looked at the filesystem:

my $jq = JQ::XS->new($untrusted, allow_includes => 0);   # croaks on include

This is decided from the program text, but it is exact rather than a guess: jq's grammar only accepts include and import in the program prologue, so that is the only place they can be, and legal programs that merely use the words elsewhere -- {include: 1}, .foo.include -- still compile.

Where those directives look

library_paths does not decide whether a directive may load anything, only where it looks, as jq's -L does:

my $jq = JQ::XS->new($prog, library_paths => ['/usr/share/myapp/jq']);

but it cannot by itself stop a program from loading anything, because jq searches the current directory when a program does not say where to look, and a program that does say -- include "x" {search:"/tmp"}; -- is not searching the library paths at all. allow_includes => 0 is the one that decides whether a program may load modules; library_paths only steers the ones it is already allowed to load.

The ~/.jq jq imports on its own

jq imports ~/.jq into every program it compiles, whether or not the program asks for it, and neither option above prevents it:

allow_includes => 0 does not see it. That option reads the program text, and this import is not in the program text -- jq's linker prepends it to every program on its way to being compiled.
library_paths does not apply to it. The import carries its own search path, the home directory, and jq consults JQ_LIBRARY_PATH only for an import that names no path of its own. An empty library_paths is simply not consulted.

So whatever ~/.jq defines is in scope for the program, and can shadow builtins for it. The import is marked optional, so it is silent when there is no such file, which is why this tends to go unnoticed on hosts that do not happen to have one.

jq locates that file through $HOME, and skips the import when $HOME is unset. Take HOME out of the environment for the call and nothing is imported:

my $jq = do {
    delete local $ENV{HOME};
    JQ::XS->new($prog);
};

Only new() has to be inside that scope. The import is resolved as the program is compiled and is then fixed in the compiled program, so restoring HOME before you run anything neither unloads a ~/.jq that was already imported nor lets one in afterwards. Nothing else in jq reads $HOME, and unsetting it changes nothing else about how a program runs.

Use delete local, not local $ENV{HOME} = ''. An empty HOME is still a home directory as far as jq is concerned, one whose ~/.jq is /.jq: the import is skipped only because that file usually does not exist, not because jq declined to look for it. delete local is what makes it not look.

On anything but Windows, HOME is the only variable jq consults. On Windows it falls back to USERPROFILE, and then to HOMEDRIVE and HOMEPATH, when HOME is unset, so all of them have to go:

delete local @ENV{qw(HOME USERPROFILE HOMEDRIVE HOMEPATH)};

Loading nothing at all

Together, for a program you did not write:

my $jq = do {
    delete local $ENV{HOME};
    JQ::XS->new($untrusted, allow_includes => 0);
};

library_paths does not appear because it does not need to: it already defaults to empty, and with includes refused there is nothing left to steer.

library_paths(\@dirs)

Without an argument, returns the current module search path as an arrayref. With one, replaces it and returns the object.

Setting it after construction does not retroactively change what the compiled program included; it is the modulemeta builtin and jq's $__loc__ machinery that read it later.

RESTRICTING ENVIRONMENT ACCESS

A jq program can read the process environment two different ways, and they are independent of each other:

$ENV is resolved while the program is compiled. jq's compiler rewrites every reference to $ENV that nothing else has already bound into a constant built by walking the process's environment -- the same object jq -n '$ENV' prints on the command line.
env is an ordinary builtin function. It reads the environment itself, fresh, each time it runs.

Neither is behind an attribute or a flag the way "RESTRICTING MODULE LOADING" is: libjq offers no switch for either, and JQ::XS adds no option of its own. What follows is something you do to the program text yourself, before it is compiled, rather than anything the module does for you.

Shadowing both

jq only substitutes the real environment for $ENV when nothing has bound that name already, and env is only special until something earlier in the program redefines it -- an ordinary def shadows a builtin exactly as it would shadow any other definition, for everything lexically after it. Put both ahead of a program and neither name reaches the real environment:

def env: {};
{} as $ENV | (
... the program ...
)

env and $ENV both come back {} inside the parentheses, and env.FOO / $ENV.FOO come back null, whatever FOO actually holds in the process running it.

Mind the closing paren

jq comments run to the end of the line, so the wrapper's own closing paren is easy to lose inside one if the program being wrapped ends in a # comment with no newline of its own after it:

# BUG: that ')' is inside the comment, not code
my $wrapped = 'def env: {}; {} as $ENV | (' . $program . ')';

That particular case fails safely -- jq reports a syntax error for the now-unclosed paren rather than compiling into something unintended -- but it is still a confusing way to make an otherwise fine program stop working. Give the closing paren its own line instead, so a trailing comment can only ever reach the newline before it:

my $wrapped = 'def env: {}; {} as $ENV | (' . "\n"
            . $program . "\n"
            . ')';
my $jq = JQ::XS->new($wrapped, allow_includes => 0);

Programs that use import/include

This prologue is textual, so it has to come before whatever the program does, and that only works when the program has no import or include directive of its own: jq's grammar allows those only at the very start of a program, a position this prologue has just taken for itself. Wrap a program that begins with one and jq reports a plain syntax error -- unexpected import -- rather than the clearer "not allowed" "RESTRICTING MODULE LOADING" gives, because the parse never gets far enough for that check to run.

Pairing the prologue with allow_includes => 0 costs nothing, then: such a program was never going to be allowed a directive in the first place, so the prologue rules out no case that was still open. That is the combination to reach for when the program is one you did not write, as in the example above.

For a program you do control and want both for, put the prologue after the directives rather than before them -- import "x" as m; def env: {}; {} as $ENV | ( ... ) compiles fine. Doing the same to a program you did not write means finding where its prologue ends, which is jq lexing rather than a regexp, so it is worth it only when you already know the shape of what you are wrapping.

Exposing a filtered subset instead

Rather than writing the replacement into the program text -- and having to think about escaping whatever it contains -- pass it through new()'s vars option instead, and reference it from the prologue:

my $jq = JQ::XS->new(
    'def env: $__jqxs_env; $__jqxs_env as $ENV | (' . "\n"
  . $program . "\n"
  . ')',
    vars => { __jqxs_env => { PATH => $ENV{PATH} } },  # Perl's %ENV here
);

env and $ENV now both come back {"PATH": ...} inside the program, with the rest of the process's environment out of reach -- a way to hand a filter the one or two variables it legitimately needs without handing it everything.

ATTRIBUTES

attr($name)

Returns the jq attribute $name, or undef if it was never set. jq keeps JQ_LIBRARY_PATH, JQ_ORIGIN and PROGRAM_ORIGIN here.

set_attr($name, $value)

Sets one. Attributes that affect compilation have to be set through new()'s attrs option instead, since by the time you have an object the program is already compiled.

EXECUTION FLAGS

flags($flags)

Sets the flags each run passes to jq, and returns the previous value. The only flags jq defines are the execution traces in "CONSTANTS", which write to the process's standard error.

$jq->flags(JQ_DEBUG_TRACE);

dump_disassembly($indent)

Writes the compiled program's bytecode to standard output, as jq's --debug-dump-disasm does. $indent defaults to 0.

jq prints this from C, to the process's file descriptor 1 rather than through Perl's STDOUT handle, so reopening STDOUT in Perl will not capture it; redirect the file descriptor if you need to.

FUNCTIONS

These are all exportable.

jq_version()

Returns the version of the jq compiled into this build, e.g. '1.8.2', or undef if the module was built against the libjq the operating system packages (see "EMBEDDED JQ").

use JQ::XS qw(jq_version);
print jq_version();      # 1.8.2

parse_json($text)

Parses one JSON value with jq's own parser and returns it as Perl data, with the same type mapping process() uses. Croaks if the text is not a single valid JSON value.

parse_json_stream($text)

The same, for text holding any number of JSON values one after another -- which is what the jq command line reads from a file or a pipe, and what parse_json() rejects. Returns them as a list.

my @values = parse_json_stream("1 2 {\"a\":3}");   # (1, 2, {a=>3})

to_json($data, %options)

Serializes Perl data to JSON with jq's own printer. Takes the formatting options set_output() takes, apart from raw:

print to_json($data, indent => 2, sort_keys => 1), "\n";

set_colors($spec)

Sets the colors color => 1 output uses, in the format of jq's JQ_COLORS environment variable -- a colon-separated list of ANSI SGR sequences for null, false, true, numbers, strings, arrays, objects and object keys:

set_colors('0;90:0;39:0;39:0;39:0;32:1;39:1;39:34;1');

Returns true if the specification was understood. This is a global setting inside libjq, not a per-object one, so it affects every JQ::XS object in the process.

features()

Returns a hashref describing what this build of the module can do, which under JQ_SYSTEM=1 depends on the libjq it was linked against:

embedded_jq: True if the jq engine is compiled into the module, false under JQ_SYSTEM=1. The same distinction jq_version() reports by returning undef.
stderr_cb: True if set_stderr_cb() works, i.e. if the libjq has the jq_set_stderr_cb that jq 1.7 added.

CONSTANTS

The following constants are available via @EXPORT_OK:

JQ_DEBUG_TRACE        - value 1
JQ_DEBUG_TRACE_DETAIL - value 2
JQ_DEBUG_TRACE_ALL    - value 3

They are the flags flags() accepts.

EMBEDDED JQ

From version 2.00, JQ::XS does not use the libjq the operating system packages. It ships an upstream jq release under vendor/, compiles it at install time and links it statically, so the installed module has no libjq to find at runtime and behaves the same way on every distribution regardless of the jq version that distribution ships. "jq_version()" reports which jq that is.

Building needs only a C compiler, make and a POSIX shell; jq's own release tarball is self-contained, and its bundled oniguruma is built too, so the regex builtins (test, match, capture, sub, gsub, scan, splits) work without an external library.

To link the OS libjq instead, configure with:

perl Makefile.PL JQ_SYSTEM=1

in which case "jq_version()" returns undef and the filter semantics are whatever the installed libjq implements. Everything the XS uses has been in libjq since jq 1.5 bar one: jq_set_stderr_cb, behind set_stderr_cb(), arrived in 1.7, and Makefile.PL link-probes for it -- against an older libjq the build says so and features() reports stderr_cb as false.

One caveat worth knowing before choosing that build, since it cannot be worked around from here: the jq 1.6 that RHEL 8 and Debian 11 ship corrupts its own heap in jq_teardown after a filter has called halt_error, which aborts the process when the JQ::XS object is freed. Later jq releases are fine, and so is the embedded build.

AUTHOR

James Rouzier <rouzier@gmail.com>

COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and/or modify it under the terms of the MIT license. See the LICENSE file included with this distribution.

To install JQ::XS, copy and paste the appropriate command in to your terminal.

cpanm

cpanm JQ::XS

CPAN shell

perl -MCPAN -e shell
install JQ::XS

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)