Troubleshoot data delivery after Elasticsearch upgrade | EPM-UL

If data delivery from EPM-UL fails after an Elasticsearch upgrade, check for a mapper_parsing_exception in /var/log/pbrest.log.

You'll see a log entry similar to this (though not exactly the same):

Nov  2 20:17:48 [472] 2154 Failed to put data to https://foo.pmul.net:9200/pmul-eventlog-ecs-20221102/_update/0a64a8e96362cea67ED0-A, curl_code 0 (HTTP returned error), http_code 400 (Bad Request), error - {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [beyondtrust_com.pmul.event] of type [date] in document with id '0a64a8e96362cea67ED0-A'. Preview of field's value: 'Accept'"}],"type":"mapper_parsing_exception","reason":"failed to parse field [beyondtrust_com.pmul.event] of type [date] in document with id '0a64a8e96362cea67ED0-A'. Preview of field's value: 'Accept'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [Accept] with format [strict_date_optional_time||epoch_second]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}},"status":400}

The solution is to modify the EPM-UL template, installed to /opt/pbul/elk/etc/pbelasticsearchtemplate.json by default, to increment the template_version and change the type of the event field. Changed data is shown in bold-italic below:

{
        "_meta": {
        ...
            "template_version": "<increase-by-1>"
        },
        ...
        "template": {
            "mappings": {
                "properties": {
                    "beyondtrust_com": {
                        "properties": {
                            "pmul": {
                                "properties": {
                                    "event": {"type": "keyword"},
        ...
    }

After the template is fixed, restart pblighttpd-svc. If the changed template is correct, then EPM-UL will be able to deliver events to Elasticsearch starting on the following day. The reason that events can no longer be delivered on the current day is that the EPM-UL indexing scheme generates unique indexes only for a specific day, with the current day’s index bound to the now-broken template. In the future, we anticipate implementing an approach to indexing that will incorporate, as a minimum, the template_version as part of the index name. With this capability, events sent immediately after a template change would be delivered to Elasticsearch using the new index.