Fluentd is not able to send .json extension file logs to logstash (fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-opensearch-amd64-2.0)

[alwalaupendra]

Describe the bug

Hi,
We are trying to send json logs from the file extension .json to logstash but we are getting below error.

2024-08-21 07:30:44 +0000 [warn]: #0 pattern not matched: "{"applicationName":"abc"}}"
2024-08-21 07:30:44 +0000 [warn]: #0 dump an error event: error_class=ArgumentError error="log does not exist" location=nil tag="fluent.warn" time=2024-08-21 07:30:44.830543276 +0000 record={"message"=>"pattern not matched: "{\"applicationName\":\"abc\"}}""}
2024-08-21 07:30:44.830543276 +0000 fluent.warn: {"message":"pattern not matched: "{\"applicationName\":\"abc\"}}""}

Sample logs:
{"applicationName":"abc"}

fluentd config:-

<source>
  @type tail
  path /opt/jenkins/deploymentlogs/output.json
  pos_file /var/log/td-agent/httpd-access.log.pos
  format json
  time_format %Y-%m-%dT%H:%M:%S
  tag *
  read_from_head true
  #exclude_path ["/var/log/fluent*"]
  #from_encoding UTF-8
  #encoding UTF-8
<parse>
    @type json
</parse>
</source>

<filter **>
  @type parser
  format json
  key_name log
  reserve_data true
</filter>

Could you please guide me how to send .json extension file json logs to logstash?
Thanks

To Reproduce

result is expected json logs

Expected behavior

result is expected json logs

Your Environment

- Fluentd version: fluent/fluentd-kubernetes-daemonset:v1.16.3-debian-opensearch-amd64-2.0
- Package version:
- Operating system:
- Kernel version:

Your Configuration

<source>
  @type tail
  path /opt/jenkins/deploymentlogs/output.json
  pos_file /var/log/td-agent/httpd-access.log.pos
  format json
  time_format %Y-%m-%dT%H:%M:%S
  tag *
  read_from_head true
  #exclude_path ["/var/log/fluent*"]
  #from_encoding UTF-8
  #encoding UTF-8
<parse>
    @type json
</parse>
</source>

<filter **>
  @type parser
  format json
  key_name log
  reserve_data true
</filter>

Your Error Log

2024-08-21 07:30:44 +0000 [warn]: #0 pattern not matched: "{\"applicationName\":\"abc\"}}"
2024-08-21 07:30:44 +0000 [warn]: #0 dump an error event: error_class=ArgumentError error="log does not exist" location=nil tag="fluent.warn" time=2024-08-21 07:30:44.830543276 +0000 record={"message"=>"pattern not matched: \"{\\\"applicationName\\\":\\\"abc\\\"}}\""}
2024-08-21 07:30:44.830543276 +0000 fluent.warn: {"message":"pattern not matched: \"{\\\"applicationName\\\":\\\"abc\\\"}}\""}

Additional context

No response

[daipom]

I moved this issue here.
Please confirm Contributing Guide.

[daipom]

Looks like this filter setting is wrong.
{"applicationName":"abc"} does not have the log key, but you specified key_name log.
What is the purpose of this filter setting?

<filter **>
  @type parser
  format json
  key_name log
  reserve_data true
</filter>

[daipom]

Hmm, perhaps there is no such plugin?
Is the problem that there is no good way to send data from Fluentd to Logstash?

[minsaALI]

Hi daipom,
We are not using any plugins .
Below is the configuration which we use to print logs/messages to standard output .
But we dont see anything in standard output.

<source>
  @type tail
  path /opt/jenkins/deploymentlogs/output.json
  pos_file /var/log/td-agent/httpd-access.log.pos
  format json
  time_format %Y-%m-%dT%H:%M:%S
  tag *
  read_from_head true
<parse>
    @type json
</parse>
</source>

<match **>
  @type stdout
</match>

[daipom]

Oh, I see!
It is indeed strange that stdout does not output anything.
I will check it!

[daipom]

Could you please share some sample logs in the file?

{"applicationName":"abc"}

Is this a full line?

Then I can read it by the following conf

<source>
  @type tail
  path /test/fluentd/case/gh-4612/sample.log
  pos_file /test/fluentd/case/gh-4612/pos
  tag test
  read_from_head true
  <parse>
    @type json
  </parse>
</source>

<match test>
  @type stdout
</match>

sample.log:

{"applicationName":"abc"}

Please note that you need to remove the pos file if you need to collect the logs again that are already collected.

[minsaALI]

Hi @daipom ,
The issue was with pos file only. Thanks for your inputs it works now.

[alwalaupendra]

Hi @daipom
In above case we are sending json message from application server to logstash by using fluentd. In logstash without using any json filter the fields are extracted and showing in kibana. I am suspecting the fields are extracted from fluentd itself. But i want extract the fields from logstash not fluetnd. Could please guide me how can i get json message without extracting fields from fluentd. I am sharing fluentd config below.

<source>
  @type tail
  path /opt/jenkins/deploymentlogs/output.json
  pos_file /var/log/td-agent/httpd-access.log.pos
  #format json
  time_format %Y-%m-%dT%H:%M:%S
  tag test
  read_from_head true
  #exclude_path ["/var/log/fluent*"]
  #from_encoding UTF-8
  #encoding UTF-8
#<parse>
#    @type json
#</parse>
    <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          keep_time_key true
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
      </parse>
</source>
<filter **>
  @type grep
<exclude>
    key message
    pattern /force flushing buffer events/
</exclude>
</filter>
<filter **>
  @type grep
<exclude>
    key message
    pattern /flushing all buffer forcedly/
</exclude>
</filter>