(puppetlabsGH-2815) Support plan-hierarchy lookups on the CLI

This adds a new flag `--plan-hierarchy` to the `bolt lookup` CLI
command, which will perform a lookup as if in a plan outside an apply
block rather than in an apply block. The command optionally accepts
variable definitions for plan variable interpolation in hiera config.
The flag is mutually exclusive with targetting options, and either a
targetting option or `--plan-hierarchy` are required. The command
returns the bare value of the lookup, rather than a Result object.

This also updates the parallelize test to verify that `return`
statements return to only run on SSH infrastructure, since it keeps
falsely failing on Windows.

!feature

* **Lookup hiera plan_hierarchy values from the CLI** ([puppetlabs#2815](puppetlabs#2815))

  The `bolt lookup` command now has a `--plan-hierarchy` flag that will
  lookup values from Hiera's `plan_hierarchy`.

documentation/hiera.md

@@ -46,13 +46,11 @@ data set by a user overrides the default data set by the module’s author.
 ## Look up data from the command line
 
 You can use the `bolt lookup` command and `Invoke-BoltLookup` PowerShell cmdlet
-to look up data from the command line. The `lookup` command looks up data in the
-context of a target, allowing you to interpolate target facts and variables in
-your hierarchy.
+to look up Hiera data from the command line.
 
-> **Note:** The `bolt lookup` and `Invoke-BoltLookup` commands only look up data
-> using the `hierarchy` key in the Hiera configuration file. `plan_hierarchy`
-> is not supported from the command line.
+### Lookup up data from hierarchy key
+Without additional options, the `lookup` command looks up data in the context of a target, allowing
+you to interpolate target facts and variables in your hierarchy.
 
 When you run the `bolt lookup` and `Invoke-BoltLookup` commands, Bolt first
 runs an `apply_prep` on each of the targets specified. This installs the
@@ -119,6 +117,29 @@ Successful on 2 targets: windows_target, ubuntu_target
 Ran on 2 targets
 ```
 
+### Look up data from plan_hierarchy key
+
+When passed `--plan-hierarchy`, the `lookup` command will look up data from Hiera's
+[plan_hierarchy](#outside-apply-blocks) key. This mimics performing a lookup outside an apply block
+in a Bolt plan.
+
+Because `plan_hierarchy` values are not specific to individual targets this command will perform
+just one lookup and return the bare value. The `--plan-hierarchy` option is cannot be passed in
+addition to `--targets`, `--rerun`, or `--query`.
+
+If your `plan_hierarchy` contains [interpolations from plan variables](#todo) you can pass values to
+interpolate to `lookup` like so:
+
+_\*nix shell command_
+```
+bolt lookup key --plan-hierarchy plan_var=interpolate_me
+```
+
+_PowerShell cmdlet_
+```
+Invoke-BoltLookup -Key key -PlanHierarchy plan_var=interpolation_me
+```
+
 ## Look up data in plans
 
 You can use the [Puppet `lookup()`

lib/bolt/bolt_option_parser.rb

@@ -67,7 +67,7 @@ def get_help_text(subcommand, action = nil)
         { flags: OPTIONS[:global] + %w[format],
           banner: GUIDE_HELP }
       when 'lookup'
-        { flags: ACTION_OPTS + %w[hiera-config],
+        { flags: ACTION_OPTS + %w[hiera-config plan-hierarchy],
           banner: LOOKUP_HELP }
       when 'module'
         case action
@@ -407,7 +407,7 @@ module            Manage Bolt project modules
           lookup
 
       #{colorize(:cyan, 'Usage')}
-          bolt lookup <key> {--targets TARGETS | --query QUERY | --rerun FILTER}
+          bolt lookup <key> {--targets TARGETS | --query QUERY | --rerun FILTER | --plan-hierarchy}
             [options]
 
       #{colorize(:cyan, 'Description')}
@@ -418,6 +418,7 @@ module            Manage Bolt project modules
 
       #{colorize(:cyan, 'Examples')}
           bolt lookup password --targets servers
+          bolt lookup password --plan-hierarchy variable=value
     HELP
 
     MODULE_HELP = <<~HELP
@@ -988,6 +989,11 @@ def initialize(options)
         @options[:resolve] = resolve
       end
 
+      separator "\n#{self.class.colorize(:cyan, 'Lookup options')}"
+      define('--plan-hierarchy', 'Look up a value with Hiera in the context of a specific plan.') do |_|
+        @options[:plan_hierarchy] = true
+      end
+
       separator "\n#{self.class.colorize(:cyan, 'Plan options')}"
       define('--pp', 'Create a new Puppet language plan.') do |_|
         @options[:puppet] = true

lib/bolt/cli.rb

@@ -354,12 +354,18 @@ def validate(options)
                               "the project, run '#{command} #{options[:object]}'."
       end
 
-      if options[:subcommand] != 'file' && options[:subcommand] != 'script' &&
+      if !%w[file script lookup].include?(options[:subcommand]) &&
          !options[:leftovers].empty?
         raise Bolt::CLIError,
               "Unknown argument(s) #{options[:leftovers].join(', ')}"
       end
 
+      target_opts = options.keys.select { |opt| TARGETING_OPTIONS.include?(opt) }
+      if options[:subcommand] == 'lookup' &&
+         target_opts.any? && options[:plan_hierarchy]
+        raise Bolt::CLIError, "The 'lookup' command accepts either targeting option OR --plan-hierarchy."
+      end
+
       if options[:noop] &&
          !(options[:subcommand] == 'task' && options[:action] == 'run') && options[:subcommand] != 'apply'
         raise Bolt::CLIError,
@@ -432,10 +438,11 @@ def execute(options)
       end
 
       # Initialize inventory and targets. Errors here are better to catch early.
-      # options[:target_args] will contain a string/array version of the targetting options this is passed to plans
+      # options[:target_args] will contain a string/array version of the targeting options this is passed to plans
       # options[:targets] will contain a resolved set of Target objects
       unless %w[guide module project secret].include?(options[:subcommand]) ||
-             %w[convert new show].include?(options[:action])
+             %w[convert new show].include?(options[:action]) ||
+             options[:plan_hierarchy]
         update_targets(options)
       end
 
@@ -516,7 +523,13 @@ def execute(options)
           code = Bolt::ProjectManager.new(config, outputter, pal).migrate
         end
       when 'lookup'
-        code = lookup(options[:object], options[:targets])
+        plan_vars = Hash[options[:leftovers].map { |a| a.split('=', 2) }]
+        # Validate functions verifies one of these was passed
+        if options[:targets]
+          code = lookup(options[:object], options[:targets], plan_vars)
+        elsif options[:plan_hierarchy]
+          code = plan_lookup(options[:object], plan_vars)
+        end
       when 'plan'
         case options[:action]
         when 'new'
@@ -694,10 +707,19 @@ def list_groups
       outputter.print_groups(inventory.group_names.sort, inventory.source, config.default_inventoryfile)
     end
 
+    # Looks up a value with Hiera as if in a plan outside an apply block, using
+    # provided variable values for interpolations
+    #
+    def plan_lookup(key, vars = {})
+      result = pal.plan_hierarchy_lookup(key, vars)
+      outputter.print_plan_lookup(result)
+      0
+    end
+
     # Looks up a value with Hiera, using targets as the contexts to perform the
-    # look ups in.
+    # look ups in. This should return the same value as a lookup in an apply block.
     #
-    def lookup(key, targets)
+    def lookup(key, targets, plan_vars = {})
       executor = Bolt::Executor.new(
         config.concurrency,
         analytics,
@@ -706,7 +728,7 @@ def lookup(key, targets)
         config.future
       )
 
-      executor.subscribe(outputter) if options.fetch(:format, 'human') == 'human'
+      executor.subscribe(outputter) if config.format == 'human'
       executor.subscribe(log_outputter)
       executor.publish_event(type: :plan_start, plan: nil)
 
@@ -716,7 +738,7 @@ def lookup(key, targets)
           targets,
           inventory,
           executor,
-          config.concurrency
+          plan_vars: plan_vars
         )
       end
 

lib/bolt/outputter/human.rb

@@ -460,6 +460,10 @@ def print_guide(guide, _topic)
         @stream.puts(guide)
       end
 
+      def print_plan_lookup(value)
+        @stream.puts(value)
+      end
+
       def print_module_list(module_list)
         module_list.each do |path, modules|
           if (mod = modules.find { |m| m[:internal_module_group] })

lib/bolt/outputter/json.rb

@@ -98,6 +98,10 @@ def print_guide(guide, topic)
         }.to_json)
       end
 
+      def print_plan_lookup(value)
+        @stream.puts(value.to_json)
+      end
+
       def print_puppetfile_result(success, puppetfile, moduledir)
         @stream.puts({ success: success,
                        puppetfile: puppetfile,

lib/bolt/pal.rb

@@ -163,9 +163,10 @@ def detect_project_conflict(project, environment)
     # Runs a block in a PAL script compiler configured for Bolt.  Catches
     # exceptions thrown by the block and re-raises them ensuring they are
     # Bolt::Errors since the script compiler block will squash all exceptions.
-    def in_bolt_compiler
+    def in_bolt_compiler(compiler_params: {})
       # TODO: If we always call this inside a bolt_executor we can remove this here
       setup
+      compiler_params = compiler_params.merge(set_local_facts: false)
       r = Puppet::Pal.in_tmp_environment('bolt', modulepath: full_modulepath, facts: {}) do |pal|
         # Only load the project if it a) exists, b) has a name it can be loaded with
         Puppet.override(bolt_project: @project,
@@ -174,7 +175,7 @@ def in_bolt_compiler
           # of modules, it must happen *after* we have overridden
           # bolt_project or the project will be ignored
           detect_project_conflict(@project, Puppet.lookup(:environments).get('bolt'))
-          pal.with_script_compiler(set_local_facts: false) do |compiler|
+          pal.with_script_compiler(**compiler_params) do |compiler|
             alias_types(compiler)
             register_resource_types(Puppet.lookup(:loaders)) if @resource_types
             begin
@@ -632,7 +633,19 @@ def run_plan(plan_name, params, executor, inventory = nil, pdb_client = nil, app
       Bolt::PlanResult.new(e, 'failure')
     end
 
-    def lookup(key, targets, inventory, executor, _concurrency)
+    def plan_hierarchy_lookup(key, vars = {})
+      # Do a lookup with a script compiler, which uses the 'plan_hierarchy' key in
+      # Hiera config.
+      with_puppet_settings do
+        # We want all of the setup and teardown that `in_bolt_compiler` does,
+        # but also want to pass keys to the script compiler.
+        in_bolt_compiler(compiler_params: { variables: vars }) do |compiler|
+          compiler.call_function('lookup', key)
+        end
+      end
+    end
+
+    def lookup(key, targets, inventory, executor, plan_vars: {})
       # Install the puppet-agent package and collect facts. Facts are
       # automatically added to the targets.
       in_plan_compiler(executor, inventory, nil) do |compiler|
@@ -654,17 +667,23 @@ def lookup(key, targets, inventory, executor, _concurrency)
 
         trusted = Puppet::Context::TrustedInformation.local(node).to_h
 
+        # Separate environment configuration from interpolation data the same
+        # way we do when compiling Puppet catalogs.
         env_conf = {
           modulepath: @modulepath.full_modulepath,
           facts:      target.facts,
-          variables:  target.vars
+        }
+
+        interpolations = {
+          variables:          plan_vars,
+          target_variables:   target.vars
         }
 
         with_puppet_settings do
           Puppet::Pal.in_tmp_environment(target.name, **env_conf) do |pal|
             Puppet.override(overrides) do
               Puppet.lookup(:pal_current_node).trusted_data = trusted
-              pal.with_catalog_compiler do |compiler|
+              pal.with_catalog_compiler(**interpolations) do |compiler|
                 Bolt::Result.for_lookup(target, key, compiler.call_function('lookup', key))
               rescue StandardError => e
                 Bolt::Result.from_exception(target, e)

pwsh_module/command.tests.ps1

@@ -311,5 +311,10 @@ Describe "test all bolt command examples" {
       $results = Invoke-BoltLookup -key 'key' -targets 'target1,target2'
       $results | Should -Be "bolt lookup key --targets target1,target2"
     }
+
+    It "bolt lookup key --plan-hierarchy" {
+      $results = Invoke-BoltLookup -key 'key' -PlanHierarchy
+      $results | Should -Be "bolt lookup key --plan-hierarchy"
+    }
   }
 }

spec/bolt/cli_spec.rb

@@ -170,7 +170,8 @@ def stub_config(file_content = {})
   end
 
   context 'lookup' do
-    let(:pal)     { double('pal', lookup: results) }
+    let(:pal)     { double('pal', lookup: results, plan_hierarchy_lookup: value) }
+    let(:value)   { 'value' }
     let(:results) { Bolt::ResultSet.new([]) }
 
     it 'errors without a key' do
@@ -180,7 +181,7 @@ def stub_config(file_content = {})
       )
     end
 
-    it 'errors without a targeting option' do
+    it 'errors without a targeting option or plan-hierarchy' do
       cli = Bolt::CLI.new(%w[lookup key])
 
       expect { cli.execute(cli.parse) }.to raise_error(
@@ -189,12 +190,33 @@ def stub_config(file_content = {})
       )
     end
 
-    it 'calls Bolt::PAL#lookup' do
-      allow(Bolt::PAL).to receive(:new).and_return(pal)
-      expect(pal).to receive(:lookup)
+    it 'errors with both a targeting option and plan-hierarchy' do
+      cli = Bolt::CLI.new(%w[lookup key --plan-hierarchy --rerun all])
 
-      cli = Bolt::CLI.new(%w[lookup key --targets foo])
-      cli.execute(cli.parse)
+      expect { cli.execute(cli.parse) }.to raise_error(
+        Bolt::CLIError,
+        /accepts either targeting option OR --plan-hierarchy/
+      )
+    end
+
+    context 'without plan-hierarchy' do
+      it 'calls Bolt::PAL#lookup' do
+        allow(Bolt::PAL).to receive(:new).and_return(pal)
+        expect(pal).to receive(:lookup)
+
+        cli = Bolt::CLI.new(%w[lookup key --targets foo])
+        cli.execute(cli.parse)
+      end
+    end
+
+    context 'with plan-hierarchy' do
+      it 'calls Bolt::PAL#plan_hierarchy_lookup' do
+        allow(Bolt::PAL).to receive(:new).and_return(pal)
+        expect(pal).to receive(:plan_hierarchy_lookup)
+
+        cli = Bolt::CLI.new(%w[lookup key --plan-hierarchy])
+        cli.execute(cli.parse)
+      end
     end
   end
 

spec/bolt/outputter/human_spec.rb

@@ -374,6 +374,12 @@
     expect(output.string).to eq(guide)
   end
 
+  it 'prints a plan-hierarchy lookup result' do
+    value = 'peanut butter'
+    outputter.print_plan_lookup(value)
+    expect(output.string.strip).to eq(value)
+  end
+
   it 'does not spin when spinner is set to false' do
     outputter.start_spin
     sleep(0.3)
@@ -439,12 +445,12 @@
       expect(output.string).to match(/2 total, 1 from inventory, 1 adhoc/)
     end
 
-    it 'prints suggestion to use a targetting option if one was not provided' do
+    it 'prints suggestion to use a targeting option if one was not provided' do
       outputter.print_targets(target_list, inventoryfile, nil, false)
       expect(output.string).to match(/Use the .* option to view specific targets/)
     end
 
-    it 'does not print suggestion to use a targetting option if one was provided' do
+    it 'does not print suggestion to use a targeting option if one was provided' do
       outputter.print_targets(target_list, inventoryfile, nil, true)
       expect(output.string).not_to match(/Use the .* option to view specific targets/)
     end
@@ -465,12 +471,12 @@
       }
     end
 
-    it 'prints suggestion to use a targetting option if one was not provided' do
+    it 'prints suggestion to use a targeting option if one was not provided' do
       outputter.print_target_info(target_list, inventoryfile, nil, false)
       expect(output.string).to match(/Use the .* option to view specific targets/)
     end
 
-    it 'does not print suggestion to use a targetting option if one was provided' do
+    it 'does not print suggestion to use a targeting option if one was provided' do
       outputter.print_target_info(target_list, inventoryfile, nil, true)
       expect(output.string).not_to match(/Use the .* option to view specific targets/)
     end

spec/bolt/outputter/json_spec.rb

@@ -166,6 +166,13 @@
     expect(parsed['guide']).to eq(guide)
   end
 
+  it 'prints a plan-hierarchy lookup value' do
+    value = 'peanut butter'
+    outputter.print_plan_lookup(value)
+    expect { JSON.parse(output.string) }.not_to raise_error
+    expect(output.string.strip).to eq("\"#{value}\"")
+  end
+
   context '#print_targets' do
     let(:inventoryfile) { '/path/to/inventory' }
 

spec/fixtures/hiera/hiera_interpolations.yaml

@@ -14,5 +14,8 @@ hierarchy:
   - name: "Trusted interpolations"
     path: "nodes/%{trusted.certname}.yaml"
 
+  - name: "Plan var interpolations"
+    path: "%{plan_var}.yaml"
+
   - name: "Common"
     path: "common.yaml"