File source not reading all the files

[shamj]

I am using file source as below:

sources:
file_logs:
type: file
include: ["/var/log/vector/*"]

There are 101 files in /var/log/vector . But it's reading only 32 files and skipping others. I don't see any errors or warning .
Please help.

[shamj]

Could someone please advise what I am missing on this.

[shamj]

Thank you. Same output. Noticed 32 files are read.

Also it keeps on watching those files. How can I delete those files after reading

[pront]

Maybe Vector stored some state and it decides to not read them again? See https://vector.dev/docs/reference/configuration/sources/file/#ignore_checkpoints

How can I delete those files after reading

• https://vector.dev/docs/reference/configuration/sources/file/#remove_after_secs
• https://vector.dev/docs/reference/configuration/sources/file/#file-deletion

[shamj]

Thank you. I am testing this. seeing the below errors:
Also I am seeing couple of warnings saying 2025-04-18 14:16:40 2025-04-18T21:16:40.732026Z WARN vector::internal_events::file::source: Currently ignoring file too small to fingerprint. file=/etc/vector/logs/minute=13/file.log.gz .

And don't see the data in destination. How can I make sure this file data goes to the destination and not ignored

There is one more error seen:
vector::internal_events::file::source: Internal log [Failed to glob path)

[shamj]

Tested adding ignore_checkpoints, it's not reading all the files. Please advise. Its blocking us to use the vector to read the files

[pront]

Currently ignoring file too small to fingerprint

These are tiny (probably empty) files and thus won't be collected until they grow. There might be nothing to read and forward.

Otherwise, you might be hitting one of the file source bugs: list e.g. #21319

[shamj]

Thank you. there are files but less chars and single line. These are static files. Regarding open file bug , it shows solved. which version did it got solved? I can test with that version.

[ghost]

We are using the following config and it works fine

sources:
  logs-source:
    type: file
    include:
      - /var/log/tests/**/*
    exclude:
      - /var/log/tests/**/*.gz
    fingerprint:
      strategy: device_and_inode

For cases, when there is a very high amount of logs and aggressive rotation, we might consider to adjust glob_minimum_cooldown_ms.

We also could set VECTOR_LOG = DEBUG to get more information about what is happening internally.

By the way, in your example you are using /var/log/vector/* - is that about Vector logs? And if so, did you consider to use internal_logs.

What type of files you are shipping, are they a plain text or archives (.gz)?

[shamj]

Thanks @air3ijai . Let me test this. we do have large amount of logs but no rotation. I used remove_wait_secs setting to delete the files . Default for glob_minimum_cooldown_ms is 1000. Should I increase this value. What is optimal value for this?

Regarding the logs , those are not internal logs these are application logs in s3 in compressed form. I tested the aws_s3 to read the logs in s3, But looks like it does have functionality to read the already existing s3 logs.

[ghost]

In your case, glob_minimum_cooldown_ms adjustment make no much sense and a default value is fine.

One interesting point from the docs - Compressed Files

One caveat with reading compressed files is that Vector is not able to efficiently seek into them. Rather than implement a potentially-expensive full scan as a seek mechanism, Vector currently will not attempt to make further reads from a file for which it has already stored a checkpoint in a previous run. For this reason, users should take care to allow Vector to fully process any compressed files before shutting the process down or moving the files to another location on disk.

It means, if the file was started to read and you restart the Vector, on next run it will not process these files again, even if it did not finished to read them. As it was already mention by @pront, you could use ignore_checkpoints as a workaround for such cases, but it also means that you might have a duplicated data for these files which are processed during restart.

I used remove_wait_secs setting to delete the files

Hopefully that is a typo from remove_after_secs and probably a value like 1 is suitable for your case just to solve possible edge cases related to Vector restarts.

[ghost]

@shamj

Let me test this. we do have large amount of logs but no rotation. I used remove_wait_secs setting to delete the files .

Looking at the Vector v0.47.0 release notes

Fix file source bug where known small files were not deleted after the specified remove_after_secs.