(puppetlabsGH-2303) Support alternate Forge, proxies when installing …

…modules

This adds support for specifying an alternate Forge and proxies when
installing modules using the `bolt module add|install` and
`Add|Install-BoltModule` commands. This change includes upstream changes
to the `puppetfile-resolver` library and a new config option in Bolt.

The new `module-install` config option accepts a map that configures an
alternate Forge and proxies to use when resolving and installing
modules. This option is identical to the current `puppetfile` config
option, and is only used when running the `bolt module add|install` and
`Add|Install-BoltModule` commands. This setting is available to both the
project config file, `bolt-project.yaml`, and the defaults config file,
`bolt-defaults.yaml`. It is not available to the `bolt.yaml` config
file.

This option is passed to the module installer, which in turn passes
it to both `puppetfile-resolver` and `r10k` when resolving and
installing modules, respectively.

!feature

* **Support alternate Forge and proxies when installing modules**
  ([puppetlabs#2303](puppetlabs#2303))

  Bolt now supports specifying an alternate Forge and proxies to use
  when installing modules using the `bolt module add|install` commands
  and `Add|Install-BoltModule` cmdlets. An alternate Forge and proxies
  can be configured using the new `module-install` configuration option
  in `bolt-project.yaml` and `bolt-defaults.yaml`.

documentation/bolt_known_issues.md

@@ -1,5 +1,94 @@
 # Known issues
 
+## Resolving module dependencies does not support proxies or alternate Forge
+
+When running the `bolt module add|install` commands or `Add|Install-BoltModule`
+cmdlets, Bolt does not support a configured proxy or alternate Forge when it
+resolves module dependencies, even if the `module-install` configuration
+option is set. Support for resolving module dependencies with proxies or an
+alternate Forge requires changes to one of Bolt's gem dependencies, which is
+currently in progress.
+
+While Bolt will not resolve module dependencies with proxies or an alternate
+Forge, it will respect this configuration when installing modules.
+
+If your project configures the `module-install` option, you may experience
+one of the following issues:
+
+- On a restricted network where a proxy is required, resolving modules may
+  cause Bolt to fail.
+
+- When an alternate Forge is specified, Bolt may resolve and install a
+  different set of modules than expected.
+
+### Install modules without resolving dependencies
+
+To avoid this limitation and any potential errors, you can manually edit your
+Puppetfile and install modules without resolving dependencies. This is a similar
+workflow as the `bolt puppetfile install` and `Install-BoltPuppetfileModules`
+commands.
+
+For example, if your project requires the `puppetlabs/apache` module, but
+you need to download modules hosted on an alternate Forge, you should write
+your Puppetfile manually. That Puppetfile may look similar to this:
+
+```ruby
+mod 'puppetlabs/apache', '5.7.0'
+mod 'puppetlabs/stdlib', '6.5.0'
+mod 'puppetlabs/concat', '6.3.0'
+mod 'puppetlabs/translate', '2.2.0'
+```
+
+Once you have a Puppetfile, you can install modules without resolving
+dependencies using the `no-resolve` command-line option:
+
+_\*nix command_
+
+```shell
+bolt module install --no-resolve
+```
+
+_PowerShell cmdlet_
+
+```powershell
+Install-BoltModule -NoResolve
+```
+
+If you need to add a module to your project, manually add the module and
+its dependencies to your Puppetfile and then run the above command again.
+
+### Set the HTTP proxy environment variables
+
+If you only need to configure a proxy to work around network restrictions,
+you can set the HTTP proxy environment variables when you run Bolt. For
+example, if you only configure the `proxy` option under `module-install` like
+this:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  proxy: http://myproxy.com
+```
+
+Then you can set the `HTTP_PROXY` and `HTTPS_PROXY` environment variables
+to this value when you run `bolt module add|install` or `Add|Install-BoltModule`.
+This will configure a proxy that Bolt will use when it makes requests to the
+Puppet Forge and GitHub when it resolves modules.
+
+_\*nix command_
+
+```shell
+HTTP_PROXY=http://myproxy.com HTTPS_PROXY=http://myproxy.com bolt module install
+```
+
+_PowerShell cmdlet_
+
+```powershell
+SET HTTP_PROXY=http://myproxy.com
+SET HTTPS_PROXY=http://myproxy.com
+Install-BoltModule
+```
+
 ## `facts` task fails on Windows targets with Facter 3 installed
 
 When running the `facts` task on a Windows target that has Facter 3 installed,

lib/bolt/cli.rb

@@ -487,9 +487,9 @@ def execute(options)
       when 'module'
         case options[:action]
         when 'add'
-          code = add_project_module(options[:object], config.project)
+          code = add_project_module(options[:object], config.project, config.module_install)
         when 'install'
-          code = install_project_modules(config.project, options[:force], options[:resolve])
+          code = install_project_modules(config.project, config.module_install, options[:force], options[:resolve])
         when 'generate-types'
           code = generate_types
         end
@@ -739,7 +739,7 @@ def generate_types
 
     # Installs modules declared in the project configuration file.
     #
-    def install_project_modules(project, force, resolve)
+    def install_project_modules(project, config, force, resolve)
       assert_project_file(project)
       assert_puppetfile_or_module_command(project.modules)
 
@@ -749,30 +749,49 @@ def install_project_modules(project, force, resolve)
         return 0
       end
 
+      if resolve != false && config.any?
+        @logger.warn(
+          "Detected configuration for 'module-install'. This configuration is currently "\
+          "only supported when installing modules, not when resolving module dependencies. "\
+          "For more information, see https://pup.pt/bolt-module-install"
+        )
+      end
+
+      modules   = project.modules || []
       installer = Bolt::ModuleInstaller.new(outputter, pal)
 
-      ok = installer.install(project.modules,
+      ok = installer.install(modules,
                              project.puppetfile,
                              project.managed_moduledir,
+                             config,
                              force: force,
                              resolve: resolve)
       ok ? 0 : 1
     end
 
     # Adds a single module to the project.
     #
-    def add_project_module(name, project)
+    def add_project_module(name, project, config)
       assert_project_file(project)
       assert_puppetfile_or_module_command(project.modules)
 
+      if config.any?
+        @logger.warn(
+          "Detected configuration for 'module-install'. This configuration is currently "\
+          "only supported when installing modules, not when resolving module dependencies. "\
+          "For more information, see https://pup.pt/bolt-module-install"
+        )
+      end
+
       modules   = project.modules || []
       installer = Bolt::ModuleInstaller.new(outputter, pal)
 
       ok = installer.add(name,
                          modules,
                          project.puppetfile,
                          project.managed_moduledir,
-                         project.project_file)
+                         project.project_file,
+                         config)
       ok ? 0 : 1
     end
 

lib/bolt/config.rb

@@ -56,7 +56,6 @@ def self.from_project(project, overrides = {})
                logs << { debug: "Loaded configuration from #{project.config_file}" } if File.exist?(project.config_file)
                c
              end
-
       data = load_defaults(project).push(
         filepath:     project.config_file,
         data:         conf,
@@ -283,6 +282,7 @@ def initialize(project, config_data, overrides = {})
         'concurrency'         => default_concurrency,
         'format'              => 'human',
         'log'                 => { 'console' => {} },
+        'module-install'      => {},
         'plugin-hooks'        => {},
         'plugin_hooks'        => {},
         'plugins'             => {},
@@ -408,7 +408,7 @@ def deep_clone
       end
 
       # Filter hashes to only include valid options
-      %w[apply-settings apply_settings puppetfile].each do |opt|
+      %w[apply-settings apply_settings module-install puppetfile].each do |opt|
         @data[opt] = @data[opt].slice(*OPTIONS.dig(opt, :properties).keys)
       end
     end
@@ -497,6 +497,25 @@ def validate
       unless TRANSPORT_CONFIG.include?(transport)
         raise UnknownTransportError, transport
       end
+
+      # Warn the user how they should be using the 'puppetfile' or
+      # 'module-install' config options. We don't error here since these
+      # settings can be set at the user or system level.
+      if @project.modules && puppetfile_config.any? && module_install.empty?
+        command = Bolt::Util.powershell? ? 'bolt project migrate' : 'Update-BoltProject'
+        @logs << { warn: "Detected configuration for 'puppetfile'. This setting is not "\
+                         "used when 'modules' is configured. Use 'module-install' instead. "\
+                         "To automatically update your project configuration, run '#{command}'." }
+      elsif @project.modules.nil? && puppetfile_config.empty? && module_install.any?
+        @logs << { warn: "Detected configuration for 'module-install'. This setting is not "\
+                         "used when 'modules' is not configured. Use 'puppetfile' instead." }
+      elsif @project.modules && puppetfile_config.any? && module_install.any?
+        @logs << { warn: "Detected configuration for 'puppetfile' and 'module-install'. Using "\
+                         "configuration for 'module-install' because 'modules' is also configured." }
+      elsif @project.modules.nil? && puppetfile_config.any? && module_install.any?
+        @logs << { warn: "Detected configuration for 'puppetfile' and 'module-install'. Using "\
+                         "configuration for 'puppetfile' because 'modules' is not configured." }
+      end
     end
 
     def default_inventoryfile
@@ -615,6 +634,10 @@ def transport
       @data['transport']
     end
 
+    def module_install
+      @project.module_install || @data['module-install']
+    end
+
     # Check if there is a case-insensitive match to the path
     def check_path_case(type, paths)
       return if paths.nil?

lib/bolt/config/options.rb

@@ -246,6 +246,41 @@ module Options
           _example: ["~/.puppetlabs/bolt/modules", "~/.puppetlabs/bolt/site-modules"],
           _default: ["project/modules", "project/site-modules", "project/site"]
         },
+        "module-install" => {
+          description: "Options that configure where Bolt downloads modules from. This option is only used when "\
+                       "installing modules using the `bolt module add|install` commands and "\
+                       "`Add|Install-BoltModule` cmdlets.",
+          type: Hash,
+          properties: {
+            "forge" => {
+              description: "A subsection that can have its own `proxy` setting to set an HTTP proxy for Forge "\
+                           "operations only, and a `baseurl` setting to specify a different Forge host.",
+              type: Hash,
+              properties: {
+                "baseurl" => {
+                  description: "The URL to the Forge host.",
+                  type: String,
+                  format: "uri",
+                  _example: "https://forge.example.com"
+                },
+                "proxy" => {
+                  description: "The HTTP proxy to use for Forge operations.",
+                  type: String,
+                  format: "uri",
+                  _example: "https://my-forge-proxy.com:8080"
+                }
+              },
+              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://my-forge-proxy.com:8080" }
+            },
+            "proxy" => {
+              description: "The HTTP proxy to use for Git and Forge operations.",
+              type: String,
+              format: "uri",
+              _example: "https://my-proxy.com:8080"
+            }
+          },
+          _plugin: false
+        },
         "modules" => {
           description: "A list of module dependencies for the project. Each dependency is a map of data specifying "\
                        "the module to install. To install the project's module dependencies, run the `bolt module "\
@@ -393,7 +428,8 @@ module Options
           _plugin: true
         },
         "puppetfile" => {
-          description: "A map containing options for the `bolt puppetfile install` command.",
+          description: "A map containing options for the `bolt puppetfile install` command and "\
+                       "`Install-BoltPuppetfile` cmdlet.",
           type: Hash,
           properties: {
             "forge" => {
@@ -408,19 +444,19 @@ module Options
                   _example: "https://forge.example.com"
                 },
                 "proxy" => {
-                  description: "The HTTP proxy to use for Git and Forge operations.",
+                  description: "The HTTP proxy to use for Forge operations.",
                   type: String,
                   format: "uri",
-                  _example: "https://forgeapi.example.com"
+                  _example: "https://my-forge-proxy.com:8080"
                 }
               },
-              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://forgeapi.example.com" }
+              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://my-forge-proxy.com:8080" }
             },
             "proxy" => {
               description: "The HTTP proxy to use for Git and Forge operations.",
               type: String,
               format: "uri",
-              _example: "https://forgeapi.example.com"
+              _example: "https://my-proxy.com:8080"
             }
           },
           _plugin: false
@@ -540,6 +576,7 @@ module Options
         format
         inventory-config
         log
+        module-install
         plugin-hooks
         plugin_hooks
         plugins
@@ -560,6 +597,7 @@ module Options
         inventoryfile
         log
         modulepath
+        module-install
         modules
         name
         plans