/ 
Sisimai-v5.0.0
⭐ Starred 78 / README.md

License Coverage Status Perl CPAN

What is Sisimai

Sisimai is a Perl module for analyzing RFC5322 bounce emails and generating structured data from parsed results. Sisimai is the system formerly known as bounceHammer 4, is the successor to bounceHammer.

Key features

Command line demo

The following screen shows a demonstration of Sisimai at the command line using Perl(p5-sisimai) and Ruby(rb-sisimai) version of Sisimai.

Setting Up Sisimai

System requirements

More details about system requirements are available at Sisimai | Getting Started page.

Install

From CPAN

$ cpanm --sudo Sisimai
--> Working on Sisimai
Fetching http://www.cpan.org/authors/id/A/AK/AKXLIX/Sisimai-4.25.5.tar.gz ... OK
...
1 distribution installed
$ perldoc -l Sisimai
/usr/local/lib/perl5/site_perl/5.30.0/Sisimai.pm

From GitHub

$ cd /usr/local/src
$ git clone https://github.com/sisimai/p5-sisimai.git
$ cd ./p5-sisimai
$ make install-from-local
--> Working on .
Configuring Sisimai-4.25.5 ... OK
1 distribution installed

Usage

Basic usage

Sisimai->rise() method provides feature for getting parsed data as Perl Hash reference from bounced email messages like following. Beginning with v4.25.6, new accessor origin which keeps the path to email file as a data source is available.

#! /usr/bin/env perl
use Sisimai;
my $v = Sisimai->rise('/path/to/mbox'); # or path to Maildir/

# Beginning with v4.23.0, both rise() and dump() method of Sisimai class can read bounce messages
# from variable instead of a path to mailbox
use IO::File;
my $r = '';
my $f = IO::File->new('/path/to/mbox'); # or path to Maildir/
{ local $/ = undef; $r = <$f>; $f->close }
my $v = Sisimai->rise(\$r);

# If you want to get bounce records which reason is "delivered", set "delivered" option to rise()
# method like the following:
my $v = Sisimai->rise('/path/to/mbox', 'delivered' => 1);

# Beginning with v5.0.0, sisimai does not return the reulst which "reason" is "vaction" by default.
# If you want to get bounce records which reason is "vacation", set "vacation" option to rise()
# method like the following:
my $v = Sisimai->rise('/path/to/mbox', 'vacation' => 1);

if( defined $v ) {
    for my $e ( @$v ) {
        print ref $e;                   # Sisimai::Data
        print ref $e->recipient;        # Sisimai::Address
        print ref $e->timestamp;        # Sisimai::Time

        print $e->addresser->address;   # shironeko@example.org # From
        print $e->recipient->address;   # kijitora@example.jp   # To
        print $e->recipient->host;      # example.jp
        print $e->deliverystatus;       # 5.1.1
        print $e->replycode;            # 550
        print $e->reason;               # userunknown
        print $e->origin;               # /var/spool/bounce/new/1740074341.eml
        print $e->hardbounce;           # 1

        my $h = $e->damn();             # Convert to HASH reference
        my $j = $e->dump('json');       # Convert to JSON string
        print $e->dump('json');         # JSON formatted bounce data
    }
}

Convert to JSON

Sisimai->dump() method provides feature for getting parsed data as JSON string from bounced email messages like the following code:

#! /usr/bin/env perl
use Sisimai;

# Get JSON string from parsed mailbox or Maildir/
my $j = Sisimai->dump('/path/to/mbox'); # or path to Maildir/
                                        # dump() is added in v4.1.27
print $j;                               # parsed data as JSON

# dump() method also accepts "delivered" option like the following code:
my $j = Sisimai->dump('/path/to/mbox', 'delivered' => 1);

Callback feature

c___ argument of Sisimai->rise and Sisimai->dump is an array reference and is a parameter to receive code references for callback feature. The first element of c___ argument is called at Sisimai::Message->parse for dealing email headers and entire message body. The second element of c___ argument is called at the end of each email file parsing. The result generated by the callback method is accessible via Sisimai::Data->catch.

[0] For email headers and the body

Callback method set in the first element of c___ is called at Sisimai::Message->parse().

#! /usr/bin/env perl
use Sisimai;
my $code = sub {
    my $args = shift;               # (*Hash)
    my $head = $args->{'headers'};  # (*Hash)  Email headers
    my $body = $args->{'message'};  # (String) Message body
    my $adds = { 'x-mailer' => '', 'queue-id' => '' };

    if( $body =~ m/^X-Postfix-Queue-ID:\s*(.+)$/m ) {
        $adds->{'queue-id'} = $1;
    }

    $adds->{'x-mailer'} = $head->{'x-mailer'} || '';
    return $adds;
};
my $data = Sisimai->rise('/path/to/mbox', 'c___' => [$code, undef]);
my $json = Sisimai->dump('/path/to/mbox', 'c___' => [$code, undef]);

print $data->[0]->catch->{'x-mailer'};    # "Apple Mail (2.1283)"
print $data->[0]->catch->{'queue-id'};    # "43f4KX6WR7z1xcMG"

[1] For each email file

Callback method set in the second element of c___ is called at Sisimai->rise() method for dealing each email file.

my $path = '/path/to/maildir';
my $code = sub {
    my $args = shift;           # (*Hash)
    my $kind = $args->{'kind'}; # (String)  Sisimai::Mail->kind
    my $mail = $args->{'mail'}; # (*String) Entire email message
    my $path = $args->{'path'}; # (String)  Sisimai::Mail->path
    my $sisi = $args->{'sisi'}; # (*Array)  List of Sisimai::Data

    for my $e ( @$sisi ) {
        # Insert custom fields into the parsed results
        $e->{'catch'} ||= {};
        $e->{'catch'}->{'size'} = length $$mail;
        $e->{'catch'}->{'kind'} = ucfirst $kind;

        if( $$mail =~ /^Return-Path: (.+)$/m ) {
            # Return-Path: <MAILER-DAEMON>
            $e->{'catch'}->{'return-path'} = $1;
        }

        # Append X-Sisimai-Parsed: header and save into other path
        my $a = sprintf("X-Sisimai-Parsed: %d\n", scalar @$sisi);
        my $p = sprintf("/path/to/another/directory/sisimai-%s.eml", $e->token);
        my $f = IO::File->new($p, 'w');
        my $v = $$mail; $v =~ s/^(From:.+)$/$a$1/m;
        print $f $v; $f->close;
    }

    # Remove the email file in Maildir/ after parsed
    unlink $path if $kind eq 'maildir';

    # Need to not return a value
};

my $list = Sisimai->rise($path, 'c___' => [undef, $code]);
print $list->[0]->{'catch'}->{'size'};          # 2202
print $list->[0]->{'catch'}->{'kind'};          # "Maildir"
print $list->[0]->{'catch'}->{'return-path'};   # "<MAILER-DAEMON>"

More information about the callback feature is available at Sisimai | How To Parse - Callback Page.

One-Liner

Beginning with Sisimai 4.1.27, Sisimai->dump() method is available and you can get parsed data as JSON using the method.

$ perl -MSisimai -lE 'print Sisimai->dump(shift)' /path/to/mbox

Output example

[{"smtpagent": "Sendmail","reason": "hasmoved","recipient": "kijitora@example.net","replycode": "","senderdomain": "example.co.jp","alias": "","timezoneoffset": "+0900","deliverystatus": "5.1.6","timestamp": 1397086485,"origin": "set-of-emails/maildir/bsd/lhost-sendmail-22.eml","catch": {"x-mailer": "","queue-id": "","sender": ""},"destination": "example.net","subject": "Nyaaaan","lhost": "localhost","rhost": "mx-s.neko.example.jp","listid": "","messageid": "0000000011111.fff0000000003@mx.example.co.jp","addresser": "shironeko@example.co.jp","action": "failed","diagnostictype": "SMTP","smtpcommand": "DATA","feedbacktype": "","token": "61b5ea94209460ac018c1a2060bdab0acce9ffed","hardbounce": 1,"diagnosticcode": "450 busy - please try later 551 not our customer 503 need RCPT command [data]"}]

Sisimai Specification

Differences between bounceHammer and Sisimai

The following table show the differences between bounceHammer 2.7.13p3 and Sisimai. More information about differences are available at Sisimai | Differences page.

| Features | bounceHammer | Sisimai | |------------------------------------------------|---------------|-------------| | System requirements(Perl) | 5.10 - 5.14 | 5.10 - 5.30 | | Command line tools | Available | N/A | | Modules for Commercial MTAs and MPSs | N/A | Included | | WebUI/API | Included | N/A | | Database schema for storing parsed bounce data | Available | N/A[1] | | Analytical precision ratio(2000 emails)[2] | 0.61 | 1.00 | | The speed of parsing email(1000 emails) | 4.24s | 1.35s[3] | | The number of detectable bounce reasons | 19 | 29 | | The number of MTA modules(parser engine) | 15 | 68 | | Parse 2 or more bounces in a single email | Only 1st rcpt | ALL | | Parse FeedBack Loop Message/ARF format mail | Unable | OK | | Classification based on recipient domain | Available | N/A | | Output format of parsed data | YAML,JSON,CSV | JSON only | | Easy to install | No | Yes | | Install using cpan, cpanm, or cpm command | N/A | OK | | Dependencies (Except core modules of Perl) | 24 modules | 2 modules | | LOC:Source lines of code | 18200 lines | 10500 lines | | The number of tests in t/, xt/ directory | 27365 tests | 311000 tests| | License | GPLv2 or Perl | 2 clause BSD| | Support Contract provided by Developer | End Of Sales | Available |

  1. Implement yourself with using DBI or any O/R Mapper you like
  2. See ./ANALYTICAL-PRECISION
  3. Xeon E5-2640 2.5GHz x 2 cores | 5000 bogomips | 1GB RAM | Perl 5.24.1

Other specification of Sisimai

Contributing

Bug report

Please use the issue tracker to report any bugs.

Emails could not be parsed

Bounce mails which could not be parsed by Sisimai are saved in the repository set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you have found any bounce email cannot be parsed using Sisimai, please add the email into the directory and send Pull-Request to this repository.

Other Information

See also

Author

@azumakuniyuki

Copyright

Copyright (C) 2014-2023 azumakuniyuki, All Rights Reserved.

License

This software is distributed under The BSD 2-Clause License.