There is a significant delay from the source file to transforms

[Zzaniu]

I found that there is a significant delay from the moment the log is written until it reaches the transforms.
conf.toml:

data_dir = "./vector-data"

[sources.gmb_error_log]
type = "file"
include = ["gmb_logs/gmb_error.log"]
read_from = "end"
glob_minimum_cooldown_ms = 1000

[transforms.add_hostname]
type = "remap"
inputs = ["gmb_error_log"]
source = '''
if match(string!(.message), r'^\[20\d{2}') {
    .message = "GMB 23 " + (format_timestamp(now(),"%F %X,%3f", "Asia/Shanghai") ?? "") + string!(.message)
} else {
    .message = "GMB " + string!(.message)
}
'''

[sinks.send_kafka]
type = "kafka"
inputs = ["add_hostname"]
bootstrap_servers = "xx.xx.xx.xx:9092"
topic = "xx.xx"
encoding.codec = "text"
sasl.enabled = true
sasl.mechanism = "PLAIN"
sasl.username = "xx"
sasl.password = "xxxxxxx"
batch.timeout_secs = 0.01
batch.max_bytes = 10240

output:

23 2025-06-12 12:30:36,072[2025-06-12 12:30:25,869-...

the 2025-06-12 12:30:36,072 is transforms add, 2025-06-12 12:30:25,869 is log file write, The two times differ by a full 11 seconds. Is there something wrong with my settings?

[ForkingFrenzy]

@Zzaniu
Try outputting to console to weed if kafka could be buffering.

[sinks.console]
type = "console"
inputs = [ "add_hostname" ]

  [sinks.console.encoding]
  codec = "json"

[Zzaniu]

@Zzaniu Try outputting to console to weed if kafka could be buffering.如果 kafka 可能缓冲，请尝试输出到 console 以除草。

[sinks.console]
type = "console"
inputs = [ "add_hostname" ]

  [sinks.console.encoding]
  codec = "json"

But I added the time in transforms, not sinks