NAME

files-to-elasticsearch.pl - A simple utility to tail a file and index each line as a document in ElasticSearch

VERSION

version 0.002

SYNOPSIS

To see available options, run:

file-to-elasticsearch.pl --help

Create a config file and run the utility:

file-to-elasticsearch.pl --config config.yaml --log4perl logging.conf --debug

This will run a single threaded POE instance that will tail the log files you've requested, performing the requested transformations and sending them to the elasticsearch cluster and index you've specified.

CONFIGURATION

The configuration file is YAML and looks like:

---
elasticsearch:
  servers: [ "localhost:9200" ]
  type: "log"
  index: "syslog-%Y.%m.%d"

tail:
  - file: '/var/log/osquery/result.log'
    index: "osquery-result-%Y.%m.%d"
    decode: json
    extract:
      - by: split
        from: name
        when: '^pack'
        into: 'pack'
        split_on: '/'
        split_parts: [ null, "name", "report" ]
    mutate:
      prune: true
      remove: [ "calendarTime", "epoch", "counter", "_raw" ]
      rename:
        unixTime: _epoch

AUTHOR

Brad Lhotsky <brad@divisionbyzero.net>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2018 by Brad Lhotsky.

This is free software, licensed under:

The (three-clause) BSD License