/ 
File-SOPS-0.003
/ File::SOPS::Metadata::Flat

NAME

File::SOPS::Metadata::Flat - the flat sops_age__list_0__map_enc metadata encoding of the ENV and INI formats

VERSION

version 0.003

SYNOPSIS

use File::SOPS::Metadata;
use File::SOPS::Metadata::Flat;

# ENV: the flat keys live among the data keys under a sops_ prefix
my $flat = File::SOPS::Metadata::Flat->new(prefix => 'sops_');

# INI: the flat keys live bare inside a [sops] section
my $flat = File::SOPS::Metadata::Flat->new;

my @pairs = $flat->flatten($meta->to_hash);
# => ([ 'sops_age__list_0__map_enc' => '-----BEGIN…\n…\n' ], …)

my $meta = File::SOPS::Metadata->from_hash($flat->unflatten(\%lines));

DESCRIPTION

The sops metadata section has two wire formats, not one. YAML and JSON carry it as the nested mapping "to_hash" in File::SOPS::Metadata produces. The ENV and INI formats have no nesting to carry it in, so sops flattens the same mapping onto single keys with a path-mangling scheme of its own:

ENV:  sops_age__list_0__map_enc=-----BEGIN AGE ENCRYPTED FILE-----\n…
      sops_lastmodified=2026-08-21T01:34:51Z

INI:  [sops]
      age__list_0__map_enc       = -----BEGIN AGE ENCRYPTED FILE-----\n…
      lastmodified               = 2026-08-21T01:34:51Z

That is a second metadata wire format and not a formatting detail, which is why it is a module rather than a helper inside a format handler: the two formats differ only in where the flat keys are put, and building it twice is how they would drift apart. The difference is "prefix" and nothing else.

This class is deliberately untyped and schema-free. It knows the mangling scheme and the escape, and nothing about what a sops section contains -- File::SOPS::Metadata owns that. Feed "flatten" the output of $meta->to_hash and feed File::SOPS::Metadata->from_hash the output of "unflatten".

The scheme, as measured

Against sops 3.13.3, both formats, with two age recipients so that __list_1__ is a real observation and not a guess:

  • Descending into a map under key K appends __map_K.

  • Descending into a list at index N appends __list_N.

  • The root field name is bare -- age, not __map_age.

  • They compose to any depth. key_groups__list_0__map_age__list_0__map_enc is what a --shamir-secret-sharing-threshold 2 document writes.

  • List indices must run from 0 without gaps. Renumbering age__list_1__ to age__list_2__ in a file sops wrote makes it refuse the document with Error while unflattening: Incomplete list, exit 1. "unflatten" croaks on the same input rather than handing back a list with a hole in it.

Empty lists vanish, and must

$meta->to_hash always emits kms, gcp_kms, azure_kv, hc_vault, age and pgp, empty arrays included, because the nested format needs the keys to exist. The flat format has no way to write an empty list, and writing one anyway breaks the file. Measured: adding sops_kms= and sops_pgp= to a document sops wrote makes sops -d fail with

'kms[0]' expected a map or struct, got "string"
'pgp[0]' expected a map or struct, got "string"

because an empty value is read as a one-element list holding an empty string, not as an empty list. So "flatten" emits nothing at all for an empty list or map, which is exactly what sops does.

Nothing is lost by it: "from_hash" in File::SOPS::Metadata defaults every one of those fields to [] when the document does not carry it, so the round trip back through this class and from_hash restores them.

Line order is cosmetic, and produced anyway

"flatten" returns ordered pairs, walking maps in sorted key order and lists in ascending index order -- which is the order sops writes, verified with eleven recipients: age__list_10__map_enc comes after age__list_9__map_enc, so the output is a structural walk and not a byte sort of the finished flat keys.

Reordering the lines changes nothing on the read side (a file with its sops_ lines reversed still decrypts, exit 0), and the metadata section is excluded from the MAC structurally, so no digest depends on this. It is matched because producing the same bytes as the reference implementation is what this distribution is for, and because a HashRef return would throw the information away for no gain.

prefix

The string every flat key carries, and the only difference between the ENV and INI encodings. sops_ for ENV, where the metadata shares one flat namespace with the document's own keys; the empty string (the default) for INI, where the [sops] section already separates them.

is_metadata_key

next unless $flat->is_metadata_key($key);

Whether a flat key belongs to the metadata section. With an empty "prefix" every key does, which is correct for INI -- the caller has already narrowed the input to the [sops] section -- and is why an ENV caller must set the prefix before handing this class a whole document.

This exists so that the ENV handler does not have to spell /^sops_/ itself. That spelling and the one "flatten" writes have to agree, and the way they stay agreeing is by being the same string in one place.

escape_value

my $line_safe = $flat->escape_value($value);

Turns a value into the single line the flat formats store. A newline becomes the two characters backslash and n, and nothing else is touched -- not a tab, not a carriage return, and not a backslash.

That is measured, not assumed. Feeding sops an unencrypted_suffix holding each character in turn and reading the bytes back off the file it writes:

value passed to sops        bytes sops wrote
a<TAB>b                     a<TAB>b        (a real tab, unescaped)
a<CR>b                      a<CR>b         (a real CR, unescaped)
a\\b                        a\\b           (backslash not doubled)
a\tb                        a\tb           (left alone)
a<LF><LF>b                  a\n\nb

The age enc block is what makes this matter: it is PEM armor, so it is almost entirely newlines, and it has to arrive back at age byte-exact or the data key does not unwrap.

The escape is lossy, deliberately. Since a backslash is not escaped on the way out and \n is not protected on the way in, a value that already contains the two characters backslash-n is indistinguishable from one containing a newline, and comes back as a newline. That is sops's behaviour and this class reproduces it rather than inventing a lossless escape sops would not read -- see docs/adr/0022.

The same escape carries ENV data values, and there the loss is fatal

Measured for data values as well as metadata -- sixteen inputs against sops 3.13.3 -- this method reproduces the ENV store's data-value writer byte for byte, which is why the ENV handler (k36) reuses it rather than growing a second escape. Only backslash-n is affected there too: a lone backslash, \t, =, #, quotes and surrounding whitespace all survive untouched. INI does not escape its data values at all -- a multi-line one goes into go-ini's triple-quote form -- so this concerns ENV only.

The difference that matters is that a data value is in the MAC and a metadata value is not, and the digest covers the value BEFORE the escape: measured, the sops_mac plaintext of a document holding a real newline is the SHA-512 of that newline, and of one holding the two characters backslash-n is the SHA-512 of backslash-n, while the two files' data lines are byte identical. So wherever the escape does not round-trip, the document says one thing and its own MAC says another: sops writes such a file with exit 0 and then refuses to read it, MAC mismatch, exit 51.

docs/adr/0030 decides that File::SOPS refuses that value when it writes an ENV document rather than reproducing it -- the one place this distribution diverges from the reference implementation's ENV escape, and it diverges by refusing a document sops cannot read either. The rule asks this pair rather than testing for a character, so that it cannot drift away from the escape it guards:

my $bytes = File::SOPS::Encrypted->value_to_bytes($leaf);
croak ...
    unless $flat->unescape_value($flat->escape_value($bytes)) eq $bytes;

A data value's bytes are not this method's job

Note the value_to_bytes in that snippet: escape the digest bytes, never the leaf. This method's leaf handling belongs to the sops section -- it maps a JSON::PP::Boolean to lowercase true/false because that is what mac_only_encrypted is written as there. A data leaf's wire bytes come from "value_to_bytes" in File::SOPS::Encrypted, the single source of truth for the value-to-bytes mapping, and its boolean spelling is titlecase True/False.

Handing a boolean data leaf straight to this method therefore produces the right thing to write and the wrong thing to digest, and those two disagreeing is a MAC mismatch with no wrong byte anywhere to point at. Only the escape is shared; the typing is not.

unescape_value

my $value = $flat->unescape_value($line_safe);

The inverse of "escape_value": the two characters backslash and n become a newline, everything else is left alone.

Non-recursive, and a preceding backslash does not escape it -- which is sops's rule and was measured by putting each spelling into sops_lastmodified and reading the value back out of Go's own parse error, which quotes the string it got:

file bytes    sops parsed
A\nB          A<LF>B
A\\nB         A\<LF>B        (a backslash, then a newline)
A\tB          A\tB           (untouched)
A\rB          A\rB           (untouched)

Perl's s/\\n/\n/g is exactly that transform, left to right, without rescanning what it substituted.

flatten

my @pairs = $flat->flatten($meta->to_hash);

Turns the nested sops mapping into the flat key/value pairs the ENV and INI formats store, as a list of two-element ArrayRefs, in the order sops writes them: maps in sorted key order, lists in ascending index order.

Keys already carry "prefix"; values are already escaped. An empty list or map contributes nothing, which is what the format requires -- see "Empty lists vanish, and must".

Croaks on a leaf it has no encoding for: a blessed object that is not a JSON::PP::Boolean, or any reference that is neither HASH nor ARRAY. The flat format cannot express those and writing something approximate would produce a document sops refuses at a distance from the cause.

unflatten

my $section = $flat->unflatten(\%lines);
my $meta    = File::SOPS::Metadata->from_hash($section);

Rebuilds the nested sops mapping from flat key/value pairs. Keys that do not carry "prefix" are skipped, so an ENV caller may pass the whole document's key/value set; an INI caller passes the [sops] section.

Croaks rather than guessing, on every input sops also refuses: a list with a gap in its indices, a __list_ whose index is not a non-negative integer, a key that needs a mapping where another key already put a list (or the other way round), and a key set twice.

Every leaf comes back a string, and one of them is a trap

The flat formats are untyped -- there is no parser here to say that 2 was a number and true a boolean, the way YAML::XS and Cpanel::JSON::XS do for the nested format. This method is faithful to that and hands back exactly what the file holds, unescaped and otherwise untouched.

mac_only_encrypted is why that matters, and it is closed one level up. Measured against sops 3.13.3: sops_mac_only_encrypted=false added to a document whose MAC covers every value decrypts fine (exit 0), and sops_mac_only_encrypted=true on the same document fails with MAC mismatch, exit 51 -- the option selects the digest. Perl's 'false' is true, so a caller reading that string as Perl would turn a document sops reads into one this library computes the wrong digest for.

k77 decided where the coercion belongs and k138 landed it, and the decision is that this method keeps doing exactly what it does. docs/adr/0035 measured it and docs/adr/0042 implemented it: sops decodes its metadata section weakly in every format, not only in the flat ones. In a nested YAML sops: section a quoted mac_only_encrypted: "false" is the boolean false and a quoted "true" is true -- strconv.ParseBool's accepted set exactly (1 t T TRUE true True / 0 f F FALSE false False, plus the empty string as false), with yes, no, on and off refused at exit 1; and shamir_threshold: "2" is the integer 2 while "false" there is refused.

So the coercion lives in "from_hash" in File::SOPS::Metadata, the one place every format's parsed section arrives -- typed or not -- and not in this method, which is the structural inverse of "flatten" and has no schema. Putting it here would have left the same quoted spelling in a YAML document still reading as Perl-true, which is the identical bug in the format that has a handler today. So this method's output is what from_hash is built to take: hand it over unchanged, strings and all.

SEE ALSO

  • File::SOPS::Metadata - the section this encodes, and its nested form

  • docs/adr/0022 - why the escape is reproduced lossy rather than fixed

  • docs/adr/0030 - why an ENV DATA value the escape cannot carry is refused

  • docs/adr/0035 - what an untyped store writes for a typed leaf, and why the metadata typing above belongs in from_hash rather than here

  • docs/adr/0042 - the decoding as it landed there, and the accepted set for each of the two fields that are not strings

SUPPORT

Issues

Please report bugs and feature requests on GitHub at https://github.com/Getty/p5-file-sops/issues.

CONTRIBUTING

Contributions are welcome! Please fork the repository and submit a pull request.

AUTHOR

Torsten Raudssus <getty@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2026 by Torsten Raudssus.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.