apply() transfers *everything* in *all* files/ and scripts/ directories, including in the root

[rlpowell]

Describe the Bug / Steps To Reproduce

I started with a completely fresh ("bolt project init") project, the only thing I changed was adding some ssh stuff to inventory.yaml so Bolt could reach my hosts.

Normal run:

root@server:/tmp/testproj# bolt apply --execute "file { '/etc/puppetlabs': ensure => present }" --targets targethost
Starting: install puppet and gather facts on targethost
Finished: install puppet and gather facts with 0 failures in 27.34 sec
Starting: apply catalog on targethost
Started on targethost...
Finished on targethost:
  changed: 0, failed: 0, unchanged: 1 skipped: 0, noop: 0
Finished: apply catalog with 0 failures in 14.26 sec
Successful on 1 target: targethost
Ran on 1 target in 41.69 sec

Add a totally unrelated file:

root@server:/tmp/testproj# mkdir files
root@server:/tmp/testproj# dd if=/dev/urandom of=files/bigfile bs=1M count=32
32+0 records in
32+0 records out
33554432 bytes (34 MB, 32 MiB) copied, 0.233184 s, 144 MB/s
root@server:/tmp/testproj# ls -lh files/
total 32M
-rw-r--r--. 1 root root 32M Mar  9 21:09 bigfile

Apply time increases by ~6 times:

root@server:/tmp/testproj# bolt apply --execute "file { '/etc/puppetlabs': ensure => present }" --targets targethost
Starting: install puppet and gather facts on targethost
Finished: install puppet and gather facts with 0 failures in 27.39 sec
Starting: apply catalog on targethost
Started on targethost...
Finished on targethost:
  changed: 0, failed: 0, unchanged: 1 skipped: 0, noop: 0
Finished: apply catalog with 0 failures in 81.75 sec
Successful on 1 target: targethost
Ran on 1 target in 1 min, 49 sec
root@server:/tmp/testproj#

If you turn trace logging on, there's a line like this:

Packing plugin /tmp/testproj/files/bigfile to testproj/files/bigfile

I don't know if this is or is not the intended behaviour, but it's sure not the behaviour I expected!

It does this for all modules, too. All my bolt modules, that is. I tried
sticking the big file in modules/foo/files/ and it still transfers.

I have no idea why, or what use or value copying all those files provides.

It may be that the intent of #1934 was to resolve it; if so, this is a documentation bug, because this plugin packing is happening on the apply() step and https://puppet.com/docs/bolt/latest/plan_functions.html#apply doesn't mention it at all. Further, there's nothing in the apply documentation that says "heads up it's going to copy everything it might take a while".

Assuming this is the intended behaviour, and the code sure makes it look that way, I have two further issues:

1. I'd like to know what I should do with large files that I intend to send to targets from my plans with upload_file()? I definitely don't want to transfer giant binaries for plans that don't use them, so where should I put such files?
2. I'd like to see documented what these files are for. Like, I searched literally every Bolt doc I could find for a way to do:

file { '/etc/foo.conf':
	source => 'some/file/on/the/server',
}

, and I found no way to do this nor any discussion or examples of doing so. If this issue is the mechanism, that's fantastic, but it needs to be documented.

Environment

root@server:/tmp/testproj# bolt --version
3.21.0
root@server:/tmp/testproj# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

It's running under docker, although obviously that shouldn't matter.

[donoghuc]

I think we could certainly look in to better warnings about this behavior. I do have a potential solution for you here though #1934 You can specify only the modules you care about ensuring are on the target for the apply.

I'm not sure I understand your question here?

I'd like to see documented what these files are for. Like, I searched literally every Bolt doc I could find for a way to do:
file { '/etc/foo.conf':
source => 'some/file/on/the/server',
}
, and I found no way to do this nor any discussion or examples of doing so. If this issue is the mechanism, that's fantastic, but it needs to be documented.

Would the feature I pointed out make that a moot point?

[rlpowell]

I already pointed out #1934 in my original post. It does, in fact, provide a workaround, like so:

$results = apply($server_target, _run_as => root, _required_modules => ['none']) {

This appears to result in no files/ directories getting checked. Which means there's a doc bug, because as I pointed out, the docs for apply don't mention this argument at all.

The question you quoted can be stated much more simply: by default, bolt apply(...) copies all the files/ and scripts/ directories out to the client. Why? In other words: What are they used for on the client? How can I, as a Bolt user, take advantage of those files being present on the client? Is there a minimal example for the value of performing this copy?

[donoghuc]

We can certainly look in to better docs for this.

Files that are copied over can be used like so:
if you have a file in my_module/files/foo.conf
You can use it in an apply block like:

file { '/etc/foo.conf':
  source => 'puppet:///modules/my_module/foo.conf'
}

[rlpowell]

Ah, that's super cool! Yeah, there's no indication of that in the Bolt docs that I could find, and I looked pretty hard.

Thanks for the help!

[rlpowell]

A few other related comments that seem worth sharing:

1. Can confirm that the source thing you showed above works; yay! However, I can't find an invocation that works for things under files/ in the root of the project directory; this for example doesn't work but I tried many variations:

root@server:/src# ls -l files/myfile
-rw-r--r--. 1 root root 4 Mar 11 20:38 files/myfile
root@server:/src# bolt apply --log-level trace --trace --verbose --execute "file { '/tmp/hi': ensure => present, source => 'puppet:///modules/myfile' }" --targets targethost

, which leaves me again wondering, what is the usefulness of copying those files out?
2. Just FYI, this doesn't affect me directly: while I am able to disable this behaviour with _required_modules in apply() blocks, I am not able to disable it at the command line; bolt apply -m none doesn't copy any of my modules' files directories, but it does copy the one in the root.
3. The following works, is more general than the puppet:/// mechanism you showed (because it works for any file on the server), and also isn't documented anywhere:

root@server:/src# bolt apply -m none --log-level trace --trace --verbose --execute "file { '/tmp/hi': ensure => present, content => file('/tmp/myfile') }" --targets targethost

Whether it being more general is a good or bad thing is certainly subject to debate. :)
Given that that works, the copy-everything-in-files/ step seems superfluous, but that's your call of course.
4. Just a side comment for posterity, but I wonder if this is what happened with issue #1459 .

[rlpowell]

One question of mine that I didn't see answered: is there, in fact, a community standard for where to store large files that only belong on some clients and are going to be distributed with upload_file()?

[github-actions]

This issue has not had activity for 60 days and will be marked as stale.
If this issue continues to have no activity for 7 days, it will be closed.

[rlpowell]

This issue absolutely still exists.

[github-actions]

This issue has not had activity for 60 days and will be marked as stale.
If this issue continues to have no activity for 7 days, it will be closed.

[rlpowell]

I do not see any of the documentation updates this issue implies.

[github-actions]

This issue has not had activity for 60 days and will be marked as stale.
If this issue continues to have no activity for 7 days, it will be closed.

[github-actions]

This issue is stale and has been closed. If you believe this is in error,
or would like the Bolt team to reconsider it, please reopen the issue.