/ 
Encode-Wide-0.07
/ Encode::Wide

NAME

Encode::Wide - Convert wide characters (Unicode, UTF-8, etc.) into ASCII-safe HTML or XML entities

VERSION

0.07

SYNOPSIS

use Encode::Wide qw(wide_to_html wide_to_xml);

# Basic HTML conversion
my $html = wide_to_html(string => "Cafe\x{E9} d\x{E9}j\x{E0} vu");
# => 'Café déjà vu'

# Basic XML conversion (numeric entities, en-dash folded to hyphen)
my $xml = wide_to_xml(string => "Cafe\x{E9} \x{2013} na\x{EF}ve");
# => 'Café - naïve'

# Preserve embedded HTML markup (keep_hrefs)
my $linked = wide_to_html(
    string     => '<a href="/menu">Caf\x{E9}</a>',
    keep_hrefs => 1,
);
# => '<a href="/menu">Caf&eacute;</a>'

# Keep apostrophes literal for JavaScript contexts (keep_apos)
my $js_safe = wide_to_html(
    string    => "it\x{2019}s na\x{EF}ve",
    keep_apos => 1,
);
# => "it\x{2019}s na&iuml;ve"   (curly apostrophe kept; i-umlaut encoded)

# Get notified about unhandled characters instead of dying silently
my $out = wide_to_html(
    string   => $untrusted,
    complain => sub { warn "Unhandled: $_[0]" },
);

# Accept a scalar reference
my $text = "na\x{EF}ve";
my $safe = wide_to_html(string => \$text);
# => 'na&iuml;ve'

DESCRIPTION

Encode::Wide converts strings that contain non-ASCII (wide) characters into pure 7-bit ASCII output suitable for embedding in HTML pages or XML documents. Every non-ASCII codepoint is replaced by the appropriate entity reference so the output can be safely placed in HTML attributes, HTML body text, or XML element content without triggering encoding errors or security issues.

Why use this module?

HTML::Entities is the obvious alternative for HTML, but it makes strict assumptions about input encoding that cause silent failures when the input arrives as raw UTF-8 bytes, already-partially-encoded entities, or a mix of both. Encode::Wide handles all three representations through a multi-pass pipeline and falls back to HTML::Entities numeric encoding for any character not explicitly listed in its tables.

For XML, XML::Entities works in the opposite direction (decoding entities, not encoding them). Encode::Wide fills that gap.

Input

Both functions accept:

  • A Perl Unicode string (the internal utf8 flag is set) - the normal case when input comes from "decode" in Encode, a database driver with pg_enable_utf8, or a source file declared use utf8.

  • A raw UTF-8 byte string - the common case when input arrives from a legacy web form or an older database driver without automatic decoding. The pipeline's raw-byte substitution pass handles this transparently.

  • A scalar reference - wide_to_html(string => \$var). The string is read from the referent; the referent is not modified.

  • Already-encoded HTML entities - e.g. &eacute; or &lt;. By default the pipeline decodes these first so they are not double-encoded. Pass keep_hrefs => 1 to suppress decoding when the input contains trusted HTML that must pass through unchanged.

Output

Both functions return a defined scalar string containing only ASCII characters (code points 0x00-0x7F). The output is safe to concatenate directly into an HTML or XML document without further escaping.

Choosing between the two functions

Use wide_to_html when writing into an HTML context (<p>, <td>, attribute values, etc.). Named entities such as &eacute; and &ndash; are used wherever possible; they are compact and human-readable in the source.

Use wide_to_xml when writing into an XML context (XHTML, RSS, Atom, custom XML schemas). Named HTML entities other than the five predefined XML entities (&amp; &lt; &gt; &apos; &quot;) are not valid in XML. This function uses only hexadecimal numeric entities (&#x0E9;), which are valid in all XML 1.0 processors. Em-dashes and en-dashes are folded to a plain ASCII hyphen because many XML consumers normalise whitespace and punctuation anyway.

EXPORT

Nothing is exported by default. Import the functions you need explicitly:

use Encode::Wide qw(wide_to_html);          # one function
use Encode::Wide qw(wide_to_html wide_to_xml);  # both

COMMON PARAMETERS

Both functions accept the following named parameters in addition to string. Pass them as a flat key-value list:

wide_to_html(string => $text, keep_hrefs => 1, complain => \&handler);
string (required)

The text to encode. May be a plain scalar or a reference to a scalar. Must be defined; passing undef causes the function to croak with a usage message.

keep_hrefs (optional, default 0)

When true, angle brackets (<, >) and double-quotes (") are not escaped, allowing embedded HTML or XML markup to survive intact.

Security note: when keep_hrefs is set, entity-decoding is also suppressed. Without this suppression, an encoded payload such as &lt;script&gt; would be decoded to <script> and then pass through unescaped, creating an XSS vector. With keep_hrefs => 1 it is the caller's responsibility to ensure that the input does not contain untrusted content that could be exploited.

complain (optional)

A code reference called with a diagnostic string when the pipeline encounters a character it cannot encode. The function still croaks with a BUG: prefix after invoking the callback - complain is for logging, not recovery.

wide_to_html(
    string   => $text,
    complain => sub {
        my ($msg) = @_;
        warn "Encode::Wide gap: $msg";
    },
);

wide_to_html

Convert a Unicode or UTF-8 string into a pure-ASCII HTML fragment. Every non-ASCII character is replaced by its named HTML entity (e.g. &eacute;) where one exists, or a hexadecimal numeric entity (e.g. &#xNNNN;) otherwise. Bare ampersands, angle brackets, and double-quotes are also escaped so the result is safe to embed in HTML body text or attribute values without further processing.

Arguments

All parameters are passed as a flat key-value list. The string key may be omitted when passing a bare positional string as the first argument.

See "COMMON PARAMETERS" for string, keep_hrefs, and complain.

keep_apos (optional, default 0)

When true, apostrophes and their typographic variants (curly single quotes U+2018, U+2019; grave accent U+0060; Windows-1252 byte 0x98) are not converted to &apos;. Useful when the result will be embedded inside a JavaScript string literal where &apos; is not valid syntax.

Returns

A defined scalar string whose every character is in the ASCII range (code points 0x00-0x7F). The empty string is returned unchanged.

EXAMPLE

use Encode::Wide qw(wide_to_html);

# Accented characters to named entities
my $out = wide_to_html(string => "na\x{EF}ve caf\x{E9}");
# => 'na&iuml;ve caf&eacute;'

# Ampersands and angle brackets are escaped
$out = wide_to_html(string => 'Price < 100 & cost > 0');
# => 'Price &lt; 100 &amp; cost &gt; 0'

# Existing entities are decoded then re-encoded (no double-encoding)
$out = wide_to_html(string => '&eacute;');
# => '&eacute;'

# keep_hrefs: HTML markup passes through; only wide chars are encoded
$out = wide_to_html(
    string     => '<a href="/m\x{E9}nu">Men\x{FC}</a>',
    keep_hrefs => 1,
);
# => '<a href="/m&eacute;nu">Men&uuml;</a>'

# keep_apos: apostrophes kept for JavaScript contexts
$out = wide_to_html(
    string    => "it\x{2019}s clich\x{E9}",
    keep_apos => 1,
);
# => "it\x{2019}s clich&eacute;"

# Scalar reference input
my $text = "caf\x{E9}";
$out = wide_to_html(string => \$text);
# => 'caf&eacute;'

MESSAGES

Usage: wide_to_html() string not set

Fatal (via croak). The string parameter was undef. Resolution: pass a defined scalar or scalar reference.

TODO: wide_to_html(<hex-tokens...>)

Warning (via carp). A character survived all three byte_map passes and the encode_entities_numeric fallback. The hex tokens in the message identify the unhandled codepoint(s). Resolution: add the character to the appropriate byte_map array, or file a bug report at https://rt.cpan.org/Public/Dist/Display.html?Name=Encode-Wide.

BUG: wide_to_html(<hex-tokens...>)

Fatal (via croak), always preceded by the TODO warning above. The same unhandled-character condition caused a hard failure. This should never occur in normal use; it indicates a gap in the character tables.

API SPECIFICATION

Input

{
    string     => { type => SCALAR | SCALARREF, required => 1, defined => 1 },
    keep_hrefs => { type => BOOLEAN, optional => 1, default => 0 },
    keep_apos  => { type => BOOLEAN, optional => 1, default => 0 },
    complain   => { type => CODEREF,  optional => 1 },
}

Output

{ type => SCALAR, constraint => sub { $_[0] !~ /[^[:ascii:]]/ } }

PSEUDOCODE

 1. Unless keep_hrefs: decode HTML entities via HTML::Entities::decode
    and the four extra named entities (&ccaron; &zcaron; &Zcaron; &Scaron;)
 2. Escape bare & not followed by a valid entity name
    (possessive ++ quantifier prevents ReDoS backtracking)
 3. Unless keep_hrefs: escape <, >, and " using %_HTML_ESCAPE (no /e eval)
 4. First byte_map pass: typographic punctuation and exclamation mark
 5. Unless keep_apos: encode apostrophe variants to &apos;
    using an alternation regex built from the apostrophe key set
 6. Early return if the string is now pure ASCII
 7. Second byte_map pass: raw UTF-8 byte sequences -> named HTML entities
 8. Third byte_map pass: Perl Unicode chars (\N{U+...}) -> named HTML entities
 9. Fallback: HTML::Entities::encode_entities_numeric for any remaining
    non-ASCII codepoints
10. If non-ASCII still remains after the fallback: invoke complain callback,
    carp a TODO warning, then croak a BUG error

wide_to_xml

Convert a Unicode or UTF-8 string into a pure-ASCII XML fragment. Every non-ASCII character is replaced by a hexadecimal numeric entity (e.g. &#x0E9;). Only numeric entities are used because named HTML entities such as &eacute; are not defined in XML 1.0 outside of XHTML with a DTD. Em-dashes and en-dashes are folded to a plain ASCII hyphen -. Bare ampersands, angle brackets, and double-quotes are escaped so the output is valid XML element content.

Arguments

All parameters are passed as a flat key-value list. The string key may be omitted when passing a bare positional string as the first argument.

See "COMMON PARAMETERS" for string, keep_hrefs, and complain. This function does not accept keep_apos.

Returns

A defined scalar string whose every character is in the ASCII range (code points 0x00-0x7F). The empty string is returned unchanged.

EXAMPLE

use Encode::Wide qw(wide_to_xml);

# Accented characters become numeric entities
my $out = wide_to_xml(string => "SURN \x{017D}ganjar");
# => 'SURN &#x17D;ganjar'

# En-dash and em-dash are folded to a plain hyphen
$out = wide_to_xml(string => "2020\x{2013}2026");
# => '2020-2026'

# Ampersands and angle brackets are XML-escaped
$out = wide_to_xml(string => 'a < b & c > 0');
# => 'a &lt; b &amp; c &gt; 0'

# keep_hrefs: XML tags pass through; wide chars are still encoded
$out = wide_to_xml(
    string     => '<item lang="fr">Caf\x{E9}</item>',
    keep_hrefs => 1,
);
# => '<item lang="fr">Caf&#x0E9;</item>'

# Scalar reference input
my $text = "caf\x{E9}";
$out = wide_to_xml(string => \$text);
# => 'caf&#x0E9;'

MESSAGES

Usage: wide_to_xml() string not set

Fatal (via croak). The string parameter was undef. Resolution: pass a defined scalar or scalar reference.

TODO: wide_to_xml(<hex-tokens...>)

Warning (via carp). A character survived all three byte_map passes. The hex tokens in the message identify the unhandled codepoint(s). Resolution: add the character to the appropriate byte_map array, or file a bug report at https://rt.cpan.org/Public/Dist/Display.html?Name=Encode-Wide.

BUG: wide_to_xml(<hex-tokens...>)

Fatal (via croak), always preceded by the TODO warning above. This should never occur in normal use; it indicates a gap in the XML character tables. Unlike wide_to_html, there is no numeric-entity fallback for XML because there is no safe generic fallback that is valid in all XML contexts.

API SPECIFICATION

Input

{
    string     => { type => SCALAR | SCALARREF, required => 1, defined => 1 },
    keep_hrefs => { type => BOOLEAN, optional => 1, default => 0 },
    complain   => { type => CODEREF,  optional => 1 },
}

Output

{ type => SCALAR, constraint => sub { $_[0] !~ /[^[:ascii:]]/ } }

PSEUDOCODE

1. Unless keep_hrefs: decode HTML entities via HTML::Entities::decode
   and the four extra named entities (&ccaron; &zcaron; &Zcaron; &Scaron;)
2. Escape bare & not followed by a valid entity name
   (possessive ++ quantifier prevents ReDoS backtracking)
3. Unless keep_hrefs: escape <, >, and " using %_HTML_ESCAPE (no /e eval)
4. First byte_map pass: curly quotes -> &quot;, dashes -> -, apostrophes
5. Early return if the string is now pure ASCII
6. Second byte_map pass: raw UTF-8 byte sequences -> numeric XML entities
7. Third byte_map pass: Perl Unicode chars (\N{U+...}) -> numeric XML entities
8. If non-ASCII still remains: invoke complain callback, carp a TODO warning,
   then croak a BUG error

SECURITY

XSS via entity decode and keep_hrefs

By default both functions call HTML::Entities::decode as the first pipeline step, normalising input like &lt;b&gt; to <b> before re-escaping it. This round-trip is safe when keep_hrefs is false because the re-escape step then converts < and > back to &lt; and &gt;.

When keep_hrefs => 1 is set, the re-escape step is skipped so that existing markup survives intact. If the decode step still ran, a malicious input such as &lt;script&gt;alert(1)&lt;/script&gt; would become the raw string <script>alert(1)</script> and pass through to the output unescaped, creating a stored XSS vector.

Fix applied in 0.07: when keep_hrefs is true, the decode step is also skipped. The pipeline treats the input as already-trusted HTML; wide characters are still encoded, but entity normalisation becomes the caller's responsibility.

ReDoS in bare-ampersand substitution

The substitution that escapes bare & characters uses a negative lookahead to distinguish bare ampersands from valid entity references. A naive backtracking quantifier inside that lookahead creates O(n^2) work for inputs such as &aaaaa...X (many word characters, no closing semicolon).

Fix applied in 0.07: the character class inside the lookahead uses a possessive quantifier [A-Za-z#0-9]++, which commits matches and prevents backtracking. Perl 5.10 or later is required, consistent with the declared MIN_PERL_VERSION.

Eval-free substitutions

All substitutions in this module use plain /g rather than /ge (evaluate replacement as Perl code). The /e flag was present in earlier versions but was unnecessary: hash lookups are value interpolation, not executable code. Removing /e eliminates a class of potential code-injection issues should a future change inadvertently expose user-controlled data in the replacement expression.

LIMITATIONS

Character coverage is hand-maintained

Both functions use explicit @byte_map tables organised into three passes (raw UTF-8 bytes, \N{U+...} named chars, literal Unicode source chars). Characters not covered by these tables fall back to HTML::Entities::encode_entities_numeric in wide_to_html, or trigger a fatal BUG: error in wide_to_xml (XML has no safe generic numeric fallback). To add a missing character, extend all three passes for the relevant function and add a regression test in t/30-basics.t.

No <script> or <style> awareness

wide_to_html encodes wide characters uniformly regardless of context. It does not detect content inside <script> or <style> blocks, so passing a complete HTML document through this function will corrupt embedded scripts and stylesheets. Feed only text fragments or attribute values, not full documents.

XML numeric entity format uses a minimal hex width

The XML pipeline outputs &#x0E9; (three hex digits with one leading zero for values below 0x100) rather than the canonical four-digit form &#x00E9;. Both representations are valid XML 1.0. Consumers that perform strict byte- level comparison of entity strings should normalise to a consistent width before comparing.

Raw binary input is not supported

The module assumes its input is either a Perl Unicode string (internal utf8 flag set) or a valid UTF-8 byte string. Passing arbitrary binary data or text in a single-byte encoding other than Latin-1 will produce incorrect output or trigger decoding errors. Decode the input with "decode" in Encode before calling these functions.

keep_hrefs shifts trust to the caller

When keep_hrefs => 1 is set, entity-decoding is suppressed and markup characters pass through unescaped. The caller must guarantee that the input does not contain untrusted content that could produce XSS output.

SEE ALSO

SUPPORT

Please report bugs and feature requests through the RT bug tracker:

https://rt.cpan.org/Public/Dist/Display.html?Name=Encode-Wide

Or by email: bug-encode-wide at rt.cpan.org

You will be notified automatically of progress on your report.

FORMAL SPECIFICATION

wide_to_html

Let S be the input string, S' the output string.

∀ c ∈ S' : ord(c) ≤ 0x7F                           (ASCII-only output)
S = ""  ⟹  S' = ""                                  (empty pass-through)
keep_hrefs = 0 ⟹ "<" ¬in; S' ∧ ">" ¬in; S' ∧ ∄ bare " in S'
keep_apos  = 0 ⟹ ∄ bare apostrophe in S'
¬∃ bare & in S'  (& appears only as part of a valid &name; or &#xNN; entity)
string = undef ⟹ croak("Usage: wide_to_html() string not set")

wide_to_xml

Let S be the input string, S' the output string.

∀ c ∈ S' : ord(c) ≤ 0x7F                           (ASCII-only output)
S = ""  ⟹  S' = ""                                  (empty pass-through)
keep_hrefs = 0 ⟹ "<" ¬in; S' ∧ ">" ¬in; S' ∧ ∄ bare " in S'
U+2013 ∈ S ⟹ "-" ∈ S' ∧ U+2013 ¬in; S'               (en-dash collapsed)
U+2014 ∈ S ⟹ "-" ∈ S' ∧ U+2014 ¬in; S'               (em-dash collapsed)
¬∃ bare & in S'  (& appears only as part of a valid &name; or &#xNN; entity)
string = undef ⟹ croak("Usage: wide_to_xml() string not set")

AUTHOR

Nigel Horne, <njh at nigelhorne.com>

LICENCE AND COPYRIGHT

Copyright 2025-2026 Nigel Horne.

This library is free software; you may redistribute it and/or modify it under the same terms as Perl itself (GPL version 2 or later).

If you use this module, please let me know at njh at nigelhorne.com.