[HCL2] Support loading a shared variable definition file

[echernyavskiy]

Given the following directory structure:

templates/
-- variables.pkr.hcl
-- service1/
---- sources.pkr.hcl
---- build.pkr.hcl
---- variables.pkr.hcl
-- service2/
---- sources.pkr.hcl
---- build.pkr.hcl
---- variables.pkr.hcl

it would be nice and DRY to be able to include shared variable definitions along with per-service ones when running packer build/validate/etc..
Something along the lines of:

packer build -var-file=./templates/variables.pkr.hcl -var="shared_value=foo" -var="service1_value=bar" ./templates/service1/

It used to be possible with JSON via multiple -var-file arguments, but not with HCL2 as it seems to treat -var-file files as variable value files and only recognizes automatically loaded variable definition files.

[echernyavskiy]

Could one of the maintainers at least comment on whether this is going to be supported or not please?

[azr]

Hey @echernyavskiy yes this is currently not possible; Have you thought of making of that file a symlink in each of the subfolders ? Did that work ?

The -var-file option allows to set variables; but not define new ones.

[azr]

In terms of supporting that feature we are currently working on bringing Packer HCL2 to feature parity with Packer JSON, we're very close here and then we'd like to split out the plugins from the repo. After these two big things we'll start on improving HCL2 even more. So hang in there 🙂 .

[omissis]

Just stumbled here as I was looking for the same functionalities. Any chance there was progress in this area? 🙂

[mikeschinkel]

I will second this request.

Even if you cannot currently add it, could you speak to how you would want it added in case someone was motivated to create a pull request?

[github-actions]

This issue has been synced to JIRA for planning.

JIRA ID: HPR-818

[mikeschinkel]

Another thought on this topic. For me, at least, it would be best to add it as some type of include directive inside the HCL file vs. on the command line so that the template can be self-contained and self-documenting.

[nywilken]

Hi @mikeschinkel thanks for the additional info. I added this issue to our JIRA board so that we can start working on improving this experience, along with the information you provided in #12020.

That said, and warnings of undeclared variables aside, I believe such a template structure should work without the need for any symlinks. This of course implies that each service template has variable declarations for each of the variables it needs to load from the shared variables fie.

As part of the work to improve the variables experience I will be sure to include some example structures of what works and what does not work. We can then use that info to iterate. I will continue working to follow up on this and other related threads.

[mikeschinkel]

@nywilken — Thanks again for the quick response and promise to follow up.

" I believe such a template structure should work without the need for any symlinks"

I am confused; how would symlinks be involved here?

BTW, something like this is what I envision that could be added to the top of an HCL file:

include {
  files = [
    "./var-declarations.pkr.hcl"
  ]
}

[nywilken]

I am confused; how would symlinks be involved here?

This is reference to a comment early up on the thread and not related to your comment about an include directive.

TBH the include directive is not also needed. This is not to say that I'm ruling it out.

But having a structure that looks something like the following would work as expected (again there are some warnings at this time).

~>  tree .
.
├── service1
│   ├── source.pkr.hcl
│   ├── variables.pkr.hcl
│   └── vars.pkrvars.hcl
├── service2
│   ├── vars.pkrvars.hcl
|   ├── variables.pkr.hcl
│   └── source.pkr.hcl
└── global-variables.pkrvars.hcl

2 directories, 6 files

~>  packer build -var-file=./global-variables.pkrvars.hcl -var-file=./service1/vars.pkrvars.hcl service1
null.example: output will be in this color.

==> null.example: Running local shell script: /var/folders/k_/m9qky03d1g5bk7z3s5qr9yg00000gp/T/packer-shell3865120384
    null.example: test
Build 'null.example' finished after 34 milliseconds 853 microseconds.

==> Wait completed after 34 milliseconds 892 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.example: Did not export anything. This is the null builder

Alternatively you can name the local variables definition file in the service directories to vars.auto.pkrvars.hcl so that they get auto loaded and only the shared variable definitions file needs to be passed at the cli.

~>  packer build -var-file=./variables.pkrvars.hcl service1
null.example: output will be in this color.

==> null.example: Running local shell script: /var/folders/k_/m9qky03d1g5bk7z3s5qr9yg00000gp/T/packer-shell3589927005
    null.example: test
Build 'null.example' finished after 129 milliseconds 136 microseconds.

==> Wait completed after 129 milliseconds 176 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.example: Did not export anything. This is the null builder

[mikeschinkel]

@nywilken — Thank you again for the detailed reply.

Also sorry about the symlink question; I was confused and thinking that I was in the comments of the PR you mentioned above and did not realize this discussion was on a different issue.

"TBH the include directive is not also needed. This is not say that I'm ruling it out."

Perchange you are leaning toward ruling it out, let me please make the case for it?

I believe there are use-cases for both:

A. Variables provided in external files, and
B. Explicitly declared statements, even it they link to another file.

The use-case for A is when you have a template that you want to reuse in multiple contexts, with different values for your variables. And of course that is a very important use-case.

The use-case for B is so the implementer can ensure that whoever is reading the template is 100% aware of which variables are intended to be required. And that "reader" could be tooling.

Use-case A and B are complementary and IMO both required, where having only one is necessary but not sufficient.

Looking at it another way, I doubt anyone at Hashicorp would need to be convinced regarding the value of Infrastructure-as-Code? If shared declarations are only provided via command-line options then in-fact you do not have complete IaC, the template's correctness can only be bound to a given command that invokes it. Without an include directive testing becomes harder, and makes template linting or any other form of automated source analysis difficult if not impossible.

So please, seriously consider giving us both A and B.

[ruslan-mogilevskiy]

Is there any movement with this feature? We need it a lot to share the 'core' HCLs across different sub-projects (in separate folders).

[MaleicAcid]

+1 really need this