Add special syslog timestamp parser that uses current year #4190
Conversation
Previously it was impossible to parse syslog timestamps without the date being reported as year 0, due to the year not being specified
danielnelson
added
the
feat
| @@ -104,6 +104,7 @@ You must capture at least one field per line. | |||
| - ts-httpd ("02/Jan/2006:15:04:05 -0700") | |||
| - ts-epoch (seconds since unix epoch, may contain decimal) | |||
| - ts-epochnano (nanoseconds since unix epoch) | |||
| - ts-syslog ("Jan 02 15:04:05", parsed time is set to the current year) | |||
There was a problem hiding this comment.
@danielnelson this is the BSD syslog timestamp format (RFC3164) not the syslog RFC5424 timestamp (basically a RFC3339 micro timestamp).
So probably you want to call it ts-bsd-syslog, and maybe also add another format for ts-syslog.
Just my 2 cents.
There was a problem hiding this comment.
My understanding is that both of these RFCs describe the wire format for syslog messages, and not how the time might be written to logfiles. Is there any documention that describes how the timestamp in, for instance, /var/log/syslog are written?
Another concern I had, and maybe you can help with, is if these dates are commonly localized?
There was a problem hiding this comment.
The timestamp depends on the format (so RFC) chosen.
For example you can configure RSYSLOG to use RSYSLOG_SyslogProtocol23Format format, which means RFC5424. With this setting /var/log/* will contains RFC3339MICRO timestamps (as per RFC5424). Without this setting it will default to the old (RFC3164) BSD syslog (and timestamp) format.
There was a problem hiding this comment.
Regarding localization. RFC5425 with RFC3339MICRO timestamps is not affected.
Rather, timestamps of syslog messages following the old RFC3164 are not localized (must be always in english) (ref).
There was a problem hiding this comment.
I think I'll leave this as ts-syslog since it pairs with the built in grok pattern SYSLOGTIMESTAMP and there isn't any reason to add a special pattern for rfc5424 style timestamps since we already have ts-rfc3339 and ts-rfc3339nano. We only need a special type here to inject the year.
| @@ -285,6 +292,16 @@ func (p *Parser) ParseLine(line string) (telegraf.Metric, error) { | |||
| } else { | |||
| timestamp = time.Unix(0, iv) | |||
| } | |||
| case SYSLOG_TIMESTAMP: | |||
| ts, err := time.ParseInLocation("Jan 02 15:04:05", v, p.loc) | |||
There was a problem hiding this comment.
Hi @danielnelson
Should we use "Jan 2 15:04:05" instead of "Jan 02 15:04:05" ?
There was a problem hiding this comment.
I think maybe it should be Jan _2 15:04:05, but I can't seem to create a testcase that fails. Do you have a example that is failing?
There was a problem hiding this comment.
If we try to parse "Sep 2 09:01:55 value=42" in test function TestSyslogTimestampParser(),
the test will show the following:
=== RUN TestSyslogTimestampParser 2018/06/22 09:36:41 E! Error parsing Sep 2 09:01:55 to time layout [SYSLOG_TIMESTAMP]: parsing time "Sep 2 09:01:55" as "Jan 02 15:04:05": cannot parse "2 09:01:55" as "02"
Does this the example count?
Thanks.
There was a problem hiding this comment.
Ahh, the test was "passing" even though the parsing failed. Here is the fix: #4334
Previously it was impossible to parse syslog timestamps without the date being reported as year 0, due to the year not being specified
…a#4190) Previously it was impossible to parse syslog timestamps without the date being reported as year 0, due to the year not being specified
Previously it was impossible to parse syslog timestamps without the date being reported as year 0, due to the year not being specified.
This pull request adds the special timestamp format:
ts-syslogthat will set the year after parsing to the current year.%{SYSLOGTIMESTAMP:timestamp:ts-syslog}closes #3379
Required for all PRs: