(puppetlabsGH-2303) Support alternate Forge, proxies when installing …

…modules This adds support for specifying an alternate Forge and proxies when installing modules using the `bolt module add|install` and `Add|Install-BoltModule` commands. This change includes upstream changes to the `puppetfile-resolver` library and a new config option in Bolt. The new `module-install` config option accepts a map that configures an alternate Forge and proxies to use when resolving and installing modules. This option is identical to the current `puppetfile` config option, and is only used when running the `bolt module add|install` and `Add|Install-BoltModule` commands. This setting is available to both the project config file, `bolt-project.yaml`, and the defaults config file, `bolt-defaults.yaml`. It is not available to the `bolt.yaml` config file. This option is passed to the module installer, which in turn passes it to both `puppetfile-resolver` and `r10k` when resolving and installing modules, respectively. !feature * **Support alternate Forge and proxies when installing modules** ([puppetlabs#2303](puppetlabs#2303)) Bolt now supports specifying an alternate Forge and proxies to use when installing modules using the `bolt module add|install` commands and `Add|Install-BoltModule` cmdlets. An alternate Forge and proxies can be configured using the new `module-install` configuration option in `bolt-project.yaml` and `bolt-defaults.yaml`.
beechtom · Nov 6, 2020 · 1a07def · 1a07def
1 parent cc8bd47
commit 1a07def
Show file tree

Hide file tree

Showing 13 changed files with 232 additions and 62 deletions.
diff --git a/Gemfile b/Gemfile
@@ -23,6 +23,11 @@ end
 # Optional paint gem for rainbow outputter
 gem "paint", "~> 2.2"
 
+# TMP: Testing for compatibility with upstream changes to puppetfile-resolver
+gem "puppetfile-resolver",
+    git: 'https://github.com/beechtom/puppetfile-resolver',
+    branch: '2303/private-forge'
+
 group(:test) do
   gem "beaker-hostgenerator"
   gem "mocha", '~> 1.4.0'

diff --git a/bolt.gemspec b/bolt.gemspec
@@ -49,7 +49,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency "net-ssh-krb", "~> 0.5"
   spec.add_dependency "orchestrator_client", "~> 0.4"
   spec.add_dependency "puppet", ">= 6.18.0", "< 6.20"
-  spec.add_dependency "puppetfile-resolver", "~> 0.4"
+  # spec.add_dependency "puppetfile-resolver", "~> 0.4"
   spec.add_dependency "puppet-resource_api", ">= 1.8.1"
   spec.add_dependency "puppet-strings", "~> 2.3"
   spec.add_dependency "r10k", "~> 3.1"

diff --git a/documentation/managing_modules.md b/documentation/managing_modules.md
@@ -305,6 +305,62 @@ message if it cannot resolve a dependency due to a version requirement.
 > documentation on [Specifying
 > versions](https://puppet.com/docs/puppet/latest/modules_metadata.html#specifying-versions).
 
+## Installing Forge modules from an alternate Forge
+
+You can configure Bolt to use a Forge other than the [Puppet
+Forge](https://forge.puppet.com) when it installs Forge modules. To configure
+Bolt to use an alternate Forge, set the `module-install` configuration option in
+either your project configuration file, `bolt-project.yaml`, or the default
+configuration file, `bolt-defaults.yaml`.
+
+To use an alterate Forge for installing Forge modules, set the `baseurl` key
+under the `forge` section of the `module-install` option:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  forge:
+    baseurl: https://forge.example.com
+```
+
+📖 **Related information**
+
+- [bolt-project.yaml options](bolt_project_reference.md#module-install)
+- [bolt-defaults.yaml options](bolt_defaults_reference.md#module-install)
+
+## Installing modules using a proxy
+
+If your workstation cannot connect directly to the internet, you can configure
+Bolt to use a proxy when it installs modules. To configure Bolt to use a proxy
+when it installs modules, set the `module-install` configuration option in
+either your project configuration file, `bolt-project.yaml`, or the default
+configuration file, `bolt-defaults.yaml`.
+
+To set a global proxy that is used for installing Forge and git modules, set
+the `proxy` key under `module-install`:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  proxy: https://proxy.com:8080
+```
+
+You can also set a proxy that is only used when installing Forge modules. To
+set a proxy for installing Forge modules, set the `proxy` key under the `forge`
+section of the `module-install` option:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  forge:
+    proxy: https://forge-proxy.com:8080
+```
+
+📖 **Related information**
+
+- [bolt-project.yaml options](bolt_project_reference.md#module-install)
+- [bolt-defaults.yaml options](bolt_defaults_reference.md#module-install)
+
 ## Compatibility with Bolt versions
 
 The new module management style is incompatible with older Bolt versions, since it sets the

diff --git a/lib/bolt/cli.rb b/lib/bolt/cli.rb
@@ -472,9 +472,9 @@ def execute(options)
       when 'module'
         case options[:action]
         when 'add'
-          code = add_project_module(options[:object], config.project)
+          code = add_project_module(options[:object], config.project, config.module_install)
         when 'install'
-          code = install_project_modules(config.project, options[:force], options[:resolve])
+          code = install_project_modules(config.project, config.module_install, options[:force], options[:resolve])
         when 'generate-types'
           code = generate_types
         end
@@ -879,7 +879,7 @@ def initialize_project
         end
 
         installer = Bolt::ModuleInstaller.new(outputter, pal)
-        installer.install(options[:modules], puppetfile, moduledir)
+        installer.install(options[:modules], puppetfile, moduledir, config.module_install)
       end
 
       # If either bolt.yaml or bolt-project.yaml exist, the user has already
@@ -900,7 +900,7 @@ def initialize_project
 
     # Installs modules declared in the project configuration file.
     #
-    def install_project_modules(project, force, resolve)
+    def install_project_modules(project, config, force, resolve)
       assert_project_file(project)
 
       unless project.modules
@@ -909,19 +909,21 @@ def install_project_modules(project, force, resolve)
         return 0
       end
 
+      modules   = project.modules || []
       installer = Bolt::ModuleInstaller.new(outputter, pal)
 
-      ok = installer.install(project.modules,
+      ok = installer.install(modules,
                              project.puppetfile,
                              project.managed_moduledir,
+                             config,
                              force: force,
                              resolve: resolve)
       ok ? 0 : 1
     end
 
     # Adds a single module to the project.
     #
-    def add_project_module(name, project)
+    def add_project_module(name, project, config)
       assert_project_file(project)
 
       modules   = project.modules || []
@@ -931,7 +933,8 @@ def add_project_module(name, project)
                          modules,
                          project.puppetfile,
                          project.managed_moduledir,
-                         project.project_file)
+                         project.project_file,
+                         config)
       ok ? 0 : 1
     end
 

diff --git a/lib/bolt/config.rb b/lib/bolt/config.rb
@@ -40,7 +40,6 @@ def self.from_project(project, overrides = {})
                logs << { debug: "Loaded configuration from #{project.config_file}" } if File.exist?(project.config_file)
                c
              end
-
       data = load_defaults(project).push(
         filepath: project.config_file,
         data: conf,
@@ -212,6 +211,7 @@ def initialize(project, config_data, overrides = {})
         'concurrency'         => default_concurrency,
         'format'              => 'human',
         'log'                 => { 'console' => {} },
+        'module-install'      => {},
         'plugin_hooks'        => {},
         'plugins'             => {},
         'puppetdb'            => {},
@@ -334,7 +334,8 @@ def deep_clone
 
       # Filter hashes to only include valid options
       @data['apply_settings'] = @data['apply_settings'].slice(*OPTIONS['apply_settings'][:properties].keys)
-      @data['puppetfile'] = @data['puppetfile'].slice(*OPTIONS['puppetfile'][:properties].keys)
+      @data['puppetfile']     = @data['puppetfile'].slice(*OPTIONS['puppetfile'][:properties].keys)
+      @data['module-install'] = @data['module-install'].slice(*OPTIONS['module-install'][:properties].keys)
     end
 
     private def normalize_log(target)
@@ -528,6 +529,10 @@ def transport
       @data['transport']
     end
 
+    def module_install
+      @project.module_install || @data['module-install']
+    end
+
     # Check if there is a case-insensitive match to the path
     def check_path_case(type, paths)
       return if paths.nil?

diff --git a/lib/bolt/config/options.rb b/lib/bolt/config/options.rb
@@ -227,6 +227,41 @@ module Options
           _example: ["~/.puppetlabs/bolt/modules", "~/.puppetlabs/bolt/site-modules"],
           _default: ["project/modules", "project/site-modules", "project/site"]
         },
+        "module-install" => {
+          description: "Options that configure where Bolt downloads modules from. This option is only used when "\
+                       "installing modules using the `bolt module add|install` commands and "\
+                       "`Add|Install-BoltModule` cmdlets.",
+          type: Hash,
+          properties: {
+            "forge" => {
+              description: "A subsection that can have its own `proxy` setting to set an HTTP proxy for Forge "\
+                           "operations only, and a `baseurl` setting to specify a different Forge host.",
+              type: Hash,
+              properties: {
+                "baseurl" => {
+                  description: "The URL to the Forge host.",
+                  type: String,
+                  format: "uri",
+                  _example: "https://forge.example.com"
+                },
+                "proxy" => {
+                  description: "The HTTP proxy to use for Forge operations.",
+                  type: String,
+                  format: "uri",
+                  _example: "https://my-forge-proxy.com:8080"
+                }
+              },
+              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://my-forge-proxy.com:8080" }
+            },
+            "proxy" => {
+              description: "The HTTP proxy to use for Git and Forge operations.",
+              type: String,
+              format: "uri",
+              _example: "https://my-proxy.com:8080"
+            }
+          },
+          _plugin: false
+        },
         "modules" => {
           description: "A list of module dependencies for the project. Each dependency is a map of data specifying "\
                        "the module to install. To install the project's module dependencies, run the `bolt module "\
@@ -360,7 +395,8 @@ module Options
           _plugin: true
         },
         "puppetfile" => {
-          description: "A map containing options for the `bolt puppetfile install` command.",
+          description: "A map containing options for the `bolt puppetfile install` command and "\
+                       "`Install-BoltPuppetfile` cmdlet.",
           type: Hash,
           properties: {
             "forge" => {
@@ -375,19 +411,19 @@ module Options
                   _example: "https://forge.example.com"
                 },
                 "proxy" => {
-                  description: "The HTTP proxy to use for Git and Forge operations.",
+                  description: "The HTTP proxy to use for Forge operations.",
                   type: String,
                   format: "uri",
-                  _example: "https://forgeapi.example.com"
+                  _example: "https://my-forge-proxy.com:8080"
                 }
               },
-              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://forgeapi.example.com" }
+              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://my-forge-proxy.com:8080" }
             },
             "proxy" => {
               description: "The HTTP proxy to use for Git and Forge operations.",
               type: String,
               format: "uri",
-              _example: "https://forgeapi.example.com"
+              _example: "https://my-proxy.com:8080"
             }
           },
           _plugin: false
@@ -505,6 +541,7 @@ module Options
         format
         inventory-config
         log
+        module-install
         plugin_hooks
         plugins
         puppetdb
@@ -523,6 +560,7 @@ module Options
         inventoryfile
         log
         modulepath
+        module-install
         modules
         name
         plans

diff --git a/lib/bolt/module_installer.rb b/lib/bolt/module_installer.rb
@@ -17,14 +17,14 @@ def initialize(outputter, pal)
 
     # Adds a single module to the project.
     #
-    def add(name, specs, puppetfile_path, moduledir, config_path)
+    def add(name, specs, puppetfile_path, moduledir, project_file, config)
       project_specs = Specs.new(specs)
 
       # Exit early if project config already includes a spec with this name.
       if project_specs.include?(name)
         @outputter.print_message(
-          "Project configuration file #{config_path} already includes specification with name "\
-          "#{name}. Nothing to do."
+          "Project configuration file #{project_file} already includes specification "\
+          "with name #{name}. Nothing to do."
         )
         return true
       end
@@ -49,28 +49,28 @@ def add(name, specs, puppetfile_path, moduledir, config_path)
 
       begin
         resolve_specs.add_specs('name' => name)
-        puppetfile = Resolver.new.resolve(resolve_specs)
+        puppetfile = Resolver.new.resolve(resolve_specs, config)
       rescue Bolt::Error
         project_specs.add_specs('name' => name)
-        puppetfile = Resolver.new.resolve(project_specs)
+        puppetfile = Resolver.new.resolve(project_specs, config)
       end
 
       # Display the diff between the existing Puppetfile and the new Puppetfile.
       print_puppetfile_diff(existing_puppetfile, puppetfile)
 
       # Add the module to the project configuration.
-      @outputter.print_action_step("Updating project configuration file at #{config_path}")
+      @outputter.print_action_step("Updating project configuration file at #{project_file}")
 
-      data = Bolt::Util.read_yaml_hash(config_path, 'project')
+      data = Bolt::Util.read_yaml_hash(project_file, 'project')
       data['modules'] ||= []
       data['modules'] << name.tr('-', '/')
 
       begin
-        File.write(config_path, data.to_yaml)
+        File.write(project_file, data.to_yaml)
       rescue SystemCallError => e
         raise Bolt::FileError.new(
           "Unable to update project configuration file: #{e.message}",
-          config
+          project_file
         )
       end
 
@@ -79,7 +79,7 @@ def add(name, specs, puppetfile_path, moduledir, config_path)
       puppetfile.write(puppetfile_path, moduledir)
 
       # Install the modules.
-      install_puppetfile(puppetfile_path, moduledir)
+      install_puppetfile(puppetfile_path, moduledir, config)
     end
 
     # Outputs a diff of an old Puppetfile and a new Puppetfile.
@@ -145,7 +145,7 @@ def print_puppetfile_diff(old, new)
 
     # Installs a project's module dependencies.
     #
-    def install(specs, path, moduledir, force: false, resolve: true)
+    def install(specs, path, moduledir, config, force: false, resolve: true)
       @outputter.print_message("Installing project modules\n\n")
 
       if resolve != false
@@ -155,7 +155,7 @@ def install(specs, path, moduledir, force: false, resolve: true)
         # and write a Puppetfile.
         if force || !path.exist?
           @outputter.print_action_step("Resolving module dependencies, this may take a moment")
-          puppetfile = Resolver.new.resolve(specs)
+          puppetfile = Resolver.new.resolve(specs, config)
 
           # We get here either through 'bolt module install' which uses the
           # managed modulepath (which isn't configurable) or through bolt
@@ -177,7 +177,7 @@ def install(specs, path, moduledir, force: false, resolve: true)
       end
 
       # Install the modules.
-      install_puppetfile(path, moduledir)
+      install_puppetfile(path, moduledir, config)
     end
 
     # Installs the Puppetfile and generates types.

diff --git a/lib/bolt/module_installer/resolver.rb b/lib/bolt/module_installer/resolver.rb
@@ -9,7 +9,7 @@ class ModuleInstaller
     class Resolver
       # Resolves module specs and returns a Puppetfile object.
       #
-      def resolve(specs)
+      def resolve(specs, config = {})
         require 'puppetfile-resolver'
 
         # Build the document model from the specs.
@@ -41,7 +41,10 @@ def resolve(specs)
             cache:                 nil,
             ui:                    nil,
             module_paths:          [],
-            allow_missing_modules: false
+            allow_missing_modules: false,
+            forge_api_url:         config.dig('forge', 'baseurl'),
+            forge_proxy:           config.dig('forge', 'proxy'),
+            proxy:                 config['proxy']
           )
         rescue StandardError => e
           raise Bolt::Error.new(e.message, 'bolt/module-resolver-error')