(puppetlabsGH-2815) Support plan-hierarchy lookups on the CLI

This adds a new flag `--plan-hierarchy` to the `bolt lookup` CLI
command, which will perform a lookup as if in a plan outside an apply
block rather than in an apply block. The command optionally accepts
variable definitions for plan variable interpolation in hiera config.
The flag is mutually exclusive with targetting options, and either a
targetting option or `--plan-hierarchy` are required. The command
returns the bare value of the lookup, rather than a Result object.

This also updates the parallelize test to verify that `return`
statements return to only run on SSH infrastructure, since it keeps
falsely failing on Windows.

!feature

* **Lookup hiera plan_hierarchy values from the CLI** ([puppetlabs#2815](puppetlabs#2815))

  The `bolt lookup` command now has a `--plan-hierarchy` flag that will
  lookup values from Hiera's `plan_hierarchy`.

documentation/hiera.md

@@ -43,82 +43,6 @@ The module layer sets default values and merge behavior for a module’s class
 parameters. The module layer comes last in Hiera’s lookup order, so environment
 data set by a user overrides the default data set by the module’s author.
 
-## Look up data from the command line
-
-You can use the `bolt lookup` command and `Invoke-BoltLookup` PowerShell cmdlet
-to look up data from the command line. The `lookup` command looks up data in the
-context of a target, allowing you to interpolate target facts and variables in
-your hierarchy.
-
-> **Note:** The `bolt lookup` and `Invoke-BoltLookup` commands only look up data
-> using the `hierarchy` key in the Hiera configuration file. `plan_hierarchy`
-> is not supported from the command line.
-
-When you run the `bolt lookup` and `Invoke-BoltLookup` commands, Bolt first
-runs an `apply_prep` on each of the targets specified. This installs the
-`puppet-agent` package on the target, collects facts, and then stores the facts
-on the target to be used in interpolations.
-
-Looking up data from the command line is particularly useful if you need to
-debug a plan that includes calls to the `lookup()` function, or if you need to
-look up target-specific data such as a password for authenticating connections
-to the target.
-
-Given the following Hiera configuration at `<PROJECT DIRECTORY>/hiera.yaml`:
-
-```yaml
-# hiera.yaml
-version: 5
-
-hierarchy:
-  - name: "Per-OS defaults"
-    path: "os/%{facts.os.name}.yaml"
-  - name: "Common data"
-    path: "common.yaml"
-```
-
-And the following data source at `<PROJECT DIRECTORY>/data/os/Windows.yaml`:
-
-```yaml
-# data/os/Windows.yaml
-password: Bolt!
-```
-
-And the following data source at `<PROJECT DIRECTORY>/data/os/Ubuntu.yaml`:
-
-```yaml
-# data/os/Ubuntu.yaml
-password: Puppet!
-```
-
-You can look up the value for the `password` key from the command line using
-facts collected from your targets:
-
-_\*nix shell command_
-
-```shell
-bolt lookup password --targets windows_target,ubuntu_target
-```
-
-_PowerShell cmdlet_
-
-```powershell
-Invoke-BoltLooup -Key 'password' -Targets 'windows_target,ubuntu_target'
-```
-
-Bolt prints the value for the key to the console:
-
-```shell
-Starting: install puppet and gather facts on windows_target, ubuntu_target
-Finished: install puppet and gather facts with 0 failures in 6.7 sec
-Finished on windows_target:
-  Bolt!
-Finished on ubuntu_target:
-  Puppet!
-Successful on 2 targets: windows_target, ubuntu_target
-Ran on 2 targets
-```
-
 ## Look up data in plans
 
 You can use the [Puppet `lookup()`
@@ -365,3 +289,104 @@ plan other_plan(
 
 Bolt would error, because `$application` is not in scope in `other_plan` and the
 interpolation in the Hiera configuration would fail.
+
+## Look up data from the command line
+
+You can use the `bolt lookup` command and `Invoke-BoltLookup` PowerShell cmdlet
+to look up Hiera data from the command line.  By default, Bolt uses the `hierarchy`
+key in your `hiera.yaml` file to look up data and performs the lookup inside an apply
+block. If you need to look up data outside of an apply block, see
+[Outside apply blocks](#outside-apply-blocks).
+
+### With target context (inside apply blocks)
+
+Without additional options, the `lookup` command looks up data in the context of a target, allowing
+you to interpolate target facts and variables in your hierarchy.
+
+When you run the `bolt lookup` and `Invoke-BoltLookup` commands, Bolt first
+runs an `apply_prep` on each of the targets specified. This installs the
+`puppet-agent` package on the target, collects facts, and then stores the facts
+on the target to be used in interpolations.
+
+Looking up data from the command line is particularly useful if you need to
+debug a plan that includes calls to the `lookup()` function, or if you need to
+look up target-specific data such as a password for authenticating connections
+to the target.
+
+Given the following Hiera configuration at `<PROJECT DIRECTORY>/hiera.yaml`:
+
+```yaml
+# hiera.yaml
+version: 5
+
+hierarchy:
+  - name: "Per-OS defaults"
+    path: "os/%{facts.os.name}.yaml"
+  - name: "Common data"
+    path: "common.yaml"
+```
+
+And the following data source at `<PROJECT DIRECTORY>/data/os/Windows.yaml`:
+
+```yaml
+# data/os/Windows.yaml
+password: Bolt!
+```
+
+And the following data source at `<PROJECT DIRECTORY>/data/os/Ubuntu.yaml`:
+
+```yaml
+# data/os/Ubuntu.yaml
+password: Puppet!
+```
+
+You can look up the value for the `password` key from the command line using
+facts collected from your targets:
+
+_\*nix shell command_
+
+```shell
+bolt lookup password --targets windows_target,ubuntu_target
+```
+
+_PowerShell cmdlet_
+
+```powershell
+Invoke-BoltLooup -Key 'password' -Targets 'windows_target,ubuntu_target'
+```
+
+Bolt prints the value for the key to the console:
+
+```shell
+Starting: install puppet and gather facts on windows_target, ubuntu_target
+Finished: install puppet and gather facts with 0 failures in 6.7 sec
+Finished on windows_target:
+  Bolt!
+Finished on ubuntu_target:
+  Puppet!
+Successful on 2 targets: windows_target, ubuntu_target
+Ran on 2 targets
+```
+
+### With plan context (outside apply blocks)
+
+To look up data from Hiera's `plan_hierarchy` key, use the `lookup` command with
+the `--plan-hierarchy` option.  This mimics performing a lookup outside an apply block
+in a Bolt plan.
+
+Because `plan_hierarchy` values are not specific to individual targets, this command performs
+just one lookup and returns the bare value. You can't use the `--plan-hierarchy` option
+with the `--targets`, `--rerun`, or `--query` options.
+
+If your `plan_hierarchy` contains [interpolations from plan
+variables](#interpolations-outside-apply-blocks), you can pass values to interpolate to `lookup`:
+
+_\*nix shell command_
+```
+bolt lookup key --plan-hierarchy plan_var=interpolate_me
+```
+
+_PowerShell cmdlet_
+```
+Invoke-BoltLookup -Key key -PlanHierarchy plan_var=interpolate_me
+```

lib/bolt/bolt_option_parser.rb

@@ -67,7 +67,7 @@ def get_help_text(subcommand, action = nil)
         { flags: OPTIONS[:global] + %w[format],
           banner: GUIDE_HELP }
       when 'lookup'
-        { flags: ACTION_OPTS + %w[hiera-config],
+        { flags: ACTION_OPTS + %w[hiera-config plan-hierarchy],
           banner: LOOKUP_HELP }
       when 'module'
         case action
@@ -407,7 +407,7 @@ module            Manage Bolt project modules
           lookup
 
       #{colorize(:cyan, 'Usage')}
-          bolt lookup <key> {--targets TARGETS | --query QUERY | --rerun FILTER}
+          bolt lookup <key> {--targets TARGETS | --query QUERY | --rerun FILTER | --plan-hierarchy}
             [options]
 
       #{colorize(:cyan, 'Description')}
@@ -418,6 +418,7 @@ module            Manage Bolt project modules
 
       #{colorize(:cyan, 'Examples')}
           bolt lookup password --targets servers
+          bolt lookup password --plan-hierarchy variable=value
     HELP
 
     MODULE_HELP = <<~HELP
@@ -988,6 +989,11 @@ def initialize(options)
         @options[:resolve] = resolve
       end
 
+      separator "\n#{self.class.colorize(:cyan, 'Lookup options')}"
+      define('--plan-hierarchy', 'Look up a value with Hiera in the context of a specific plan.') do |_|
+        @options[:plan_hierarchy] = true
+      end
+
       separator "\n#{self.class.colorize(:cyan, 'Plan options')}"
       define('--pp', 'Create a new Puppet language plan.') do |_|
         @options[:puppet] = true

lib/bolt/cli.rb

@@ -354,12 +354,18 @@ def validate(options)
                               "the project, run '#{command} #{options[:object]}'."
       end
 
-      if options[:subcommand] != 'file' && options[:subcommand] != 'script' &&
+      if !%w[file script lookup].include?(options[:subcommand]) &&
          !options[:leftovers].empty?
         raise Bolt::CLIError,
               "Unknown argument(s) #{options[:leftovers].join(', ')}"
       end
 
+      target_opts = options.keys.select { |opt| TARGETING_OPTIONS.include?(opt) }
+      if options[:subcommand] == 'lookup' &&
+         target_opts.any? && options[:plan_hierarchy]
+        raise Bolt::CLIError, "The 'lookup' command accepts either targeting option OR --plan-hierarchy."
+      end
+
       if options[:noop] &&
          !(options[:subcommand] == 'task' && options[:action] == 'run') && options[:subcommand] != 'apply'
         raise Bolt::CLIError,
@@ -432,10 +438,11 @@ def execute(options)
       end
 
       # Initialize inventory and targets. Errors here are better to catch early.
-      # options[:target_args] will contain a string/array version of the targetting options this is passed to plans
+      # options[:target_args] will contain a string/array version of the targeting options this is passed to plans
       # options[:targets] will contain a resolved set of Target objects
       unless %w[guide module project secret].include?(options[:subcommand]) ||
-             %w[convert new show].include?(options[:action])
+             %w[convert new show].include?(options[:action]) ||
+             options[:plan_hierarchy]
         update_targets(options)
       end
 
@@ -516,7 +523,13 @@ def execute(options)
           code = Bolt::ProjectManager.new(config, outputter, pal).migrate
         end
       when 'lookup'
-        code = lookup(options[:object], options[:targets])
+        plan_vars = Hash[options[:leftovers].map { |a| a.split('=', 2) }]
+        # Validate functions verifies one of these was passed
+        if options[:targets]
+          code = lookup(options[:object], options[:targets], plan_vars: plan_vars)
+        elsif options[:plan_hierarchy]
+          code = plan_lookup(options[:object], plan_vars: plan_vars)
+        end
       when 'plan'
         case options[:action]
         when 'new'
@@ -694,10 +707,19 @@ def list_groups
       outputter.print_groups(inventory.group_names.sort, inventory.source, config.default_inventoryfile)
     end
 
+    # Looks up a value with Hiera as if in a plan outside an apply block, using
+    # provided variable values for interpolations
+    #
+    def plan_lookup(key, plan_vars: {})
+      result = pal.plan_hierarchy_lookup(key, plan_vars: plan_vars)
+      outputter.print_plan_lookup(result)
+      0
+    end
+
     # Looks up a value with Hiera, using targets as the contexts to perform the
-    # look ups in.
+    # look ups in. This should return the same value as a lookup in an apply block.
     #
-    def lookup(key, targets)
+    def lookup(key, targets, plan_vars: {})
       executor = Bolt::Executor.new(
         config.concurrency,
         analytics,
@@ -706,7 +728,7 @@ def lookup(key, targets)
         config.future
       )
 
-      executor.subscribe(outputter) if options.fetch(:format, 'human') == 'human'
+      executor.subscribe(outputter) if config.format == 'human'
       executor.subscribe(log_outputter)
       executor.publish_event(type: :plan_start, plan: nil)
 
@@ -716,7 +738,7 @@ def lookup(key, targets)
           targets,
           inventory,
           executor,
-          config.concurrency
+          plan_vars: plan_vars
         )
       end
 

lib/bolt/outputter/human.rb

@@ -460,6 +460,10 @@ def print_guide(guide, _topic)
         @stream.puts(guide)
       end
 
+      def print_plan_lookup(value)
+        @stream.puts(value)
+      end
+
       def print_module_list(module_list)
         module_list.each do |path, modules|
           if (mod = modules.find { |m| m[:internal_module_group] })

lib/bolt/outputter/json.rb

@@ -98,6 +98,10 @@ def print_guide(guide, topic)
         }.to_json)
       end
 
+      def print_plan_lookup(value)
+        @stream.puts(value.to_json)
+      end
+
       def print_puppetfile_result(success, puppetfile, moduledir)
         @stream.puts({ success: success,
                        puppetfile: puppetfile,