logstash - take 2 - filter to send messages from IntelMQ/python/redis to ELK
1
vote
1
answer
255
views
Following up on the heels of this question, https://stackoverflow.com/questions/40768603/logstash-trying-to-make-sense-of-strings-passed-by-intelmq-in-elasticsearch I am trying to create a refine/create a filter to receive messages from logstash to kibana.
Whilst the original requirements and answer are almost to speck, some new bots added to IntelMQ now put spaces on the fields. Obviously, they break completely the filters, and worse yet create spurious new fields and date in Elastic Search.
I also have found out the solution in the referred thread does take well in account the beginning and end of strings.
The strings itself is similar to:
{u'feed': u'openbl', u'reported_source_ip': u'115.79.215.79', u'source_cymru_cc': u'VN', u'source_time': u'2016-06-25T11:15:14+00:00', u'feed_url': u'http://www.openbl.org/lists/date_all.txt ', u'taxonomy': u'Other', u'observation_time': u'2016-11-20T22:51:25', u'source_ip': u'115.79.215.79', u'source_registry': u'apnic', u'source_allocated': u'2008-07-17', u'source_bgp_prefix': u'115.79.192.0/19', u'type': u'blacklist', u'source_as_name': u'VIETEL-AS-AP Viettel Corporation, VN', u'source_asn':u'7552'}
What to do?
Asked by Rui F Ribeiro
(57882 rep)
Dec 1, 2016, 02:46 PM
Last activity: Nov 2, 2018, 03:34 PM
Last activity: Nov 2, 2018, 03:34 PM