(puppetlabsGH-2303) Support alternate Forge, proxies when installing …

…modules This adds support for specifying an alternate Forge and proxies when installing modules using the `bolt module add|install` and `Add|Install-BoltModule` commands. This change includes upstream changes to the `puppetfile-resolver` library and a new config option in Bolt. The new `module-install` config option accepts a map that configures an alternate Forge and proxies to use when resolving and installing modules. This option is identical to the current `puppetfile` config option, and is only used when running the `bolt module add|install` and `Add|Install-BoltModule` commands. This setting is available to both the project config file, `bolt-project.yaml`, and the defaults config file, `bolt-defaults.yaml`. It is not available to the `bolt.yaml` config file. This option is passed to the module installer, which in turn passes it to both `puppetfile-resolver` and `r10k` when resolving and installing modules, respectively. !feature * **Support alternate Forge and proxies when installing modules** ([puppetlabs#2303](puppetlabs#2303)) Bolt now supports specifying an alternate Forge and proxies to use when installing modules using the `bolt module add|install` commands and `Add|Install-BoltModule` cmdlets. An alternate Forge and proxies can be configured using the new `module-install` configuration option in `bolt-project.yaml` and `bolt-defaults.yaml`.
beechtom · Nov 6, 2020 · 824fd2e · 824fd2e
1 parent cc8bd47
commit 824fd2e
Show file tree

Hide file tree

Showing 13 changed files with 243 additions and 73 deletions.
diff --git a/Gemfile b/Gemfile
@@ -23,6 +23,11 @@ end
 # Optional paint gem for rainbow outputter
 gem "paint", "~> 2.2"
 
+# TMP: Testing for compatibility with upstream changes to puppetfile-resolver
+gem "puppetfile-resolver",
+    git: 'https://github.com/beechtom/puppetfile-resolver',
+    branch: '2303/private-forge'
+
 group(:test) do
   gem "beaker-hostgenerator"
   gem "mocha", '~> 1.4.0'

diff --git a/bolt.gemspec b/bolt.gemspec
@@ -49,7 +49,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency "net-ssh-krb", "~> 0.5"
   spec.add_dependency "orchestrator_client", "~> 0.4"
   spec.add_dependency "puppet", ">= 6.18.0", "< 6.20"
-  spec.add_dependency "puppetfile-resolver", "~> 0.4"
+  # spec.add_dependency "puppetfile-resolver", "~> 0.4"
   spec.add_dependency "puppet-resource_api", ">= 1.8.1"
   spec.add_dependency "puppet-strings", "~> 2.3"
   spec.add_dependency "r10k", "~> 3.1"

diff --git a/documentation/managing_modules.md b/documentation/managing_modules.md
@@ -305,6 +305,62 @@ message if it cannot resolve a dependency due to a version requirement.
 > documentation on [Specifying
 > versions](https://puppet.com/docs/puppet/latest/modules_metadata.html#specifying-versions).
 
+## Installing Forge modules from an alternate Forge
+
+You can configure Bolt to use a Forge other than the [Puppet
+Forge](https://forge.puppet.com) when it installs Forge modules. To configure
+Bolt to use an alternate Forge, set the `module-install` configuration option in
+either your project configuration file, `bolt-project.yaml`, or the default
+configuration file, `bolt-defaults.yaml`.
+
+To use an alterate Forge for installing Forge modules, set the `baseurl` key
+under the `forge` section of the `module-install` option:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  forge:
+    baseurl: https://forge.example.com
+```
+
+📖 **Related information**
+
+- [bolt-project.yaml options](bolt_project_reference.md#module-install)
+- [bolt-defaults.yaml options](bolt_defaults_reference.md#module-install)
+
+## Installing modules using a proxy
+
+If your workstation cannot connect directly to the internet, you can configure
+Bolt to use a proxy when it installs modules. To configure Bolt to use a proxy
+when it installs modules, set the `module-install` configuration option in
+either your project configuration file, `bolt-project.yaml`, or the default
+configuration file, `bolt-defaults.yaml`.
+
+To set a global proxy that is used for installing Forge and git modules, set
+the `proxy` key under `module-install`:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  proxy: https://proxy.com:8080
+```
+
+You can also set a proxy that is only used when installing Forge modules. To
+set a proxy for installing Forge modules, set the `proxy` key under the `forge`
+section of the `module-install` option:
+
+```yaml
+# bolt-project.yaml
+module-install:
+  forge:
+    proxy: https://forge-proxy.com:8080
+```
+
+📖 **Related information**
+
+- [bolt-project.yaml options](bolt_project_reference.md#module-install)
+- [bolt-defaults.yaml options](bolt_defaults_reference.md#module-install)
+
 ## Compatibility with Bolt versions
 
 The new module management style is incompatible with older Bolt versions, since it sets the

diff --git a/lib/bolt/cli.rb b/lib/bolt/cli.rb
@@ -472,9 +472,9 @@ def execute(options)
       when 'module'
         case options[:action]
         when 'add'
-          code = add_project_module(options[:object], config.project)
+          code = add_project_module(options[:object], config.project, config.module_install)
         when 'install'
-          code = install_project_modules(config.project, options[:force], options[:resolve])
+          code = install_project_modules(config.project, config.module_install, options[:force], options[:resolve])
         when 'generate-types'
           code = generate_types
         end
@@ -849,22 +849,22 @@ def initialize_project
         end
       end
 
-      project    = Pathname.new(dir)
-      old_config = project + 'bolt.yaml'
-      config     = project + 'bolt-project.yaml'
-      puppetfile = project + 'Puppetfile'
-      moduledir  = project + 'modules'
+      project      = Pathname.new(dir)
+      config_file  = project + 'bolt.yaml'
+      project_file = project + 'bolt-project.yaml'
+      puppetfile   = project + 'Puppetfile'
+      moduledir    = project + 'modules'
 
       # Warn the user if the project directory already exists. We don't error
       # here since users might not have installed any modules yet. If both
       # bolt.yaml and bolt-project.yaml exist, this will just warn about
       # bolt-project.yaml and subsequent Bolt actions will warn about both files
       # existing.
-      if config.exist?
+      if project_file.exist?
         @logger.warn "Found existing project directory at #{project}. Skipping file creation."
-      elsif old_config.exist?
-        @logger.warn "Found existing #{old_config.basename} at #{project}. "\
-                    "#{old_config.basename} is deprecated, please rename to #{config.basename}."
+      elsif config_file.exist?
+        @logger.warn "Found existing #{config_file.basename} at #{project}. "\
+                    "#{config_file.basename} is deprecated, please rename to #{project_file.basename}."
       end
 
       # If modules were specified, first check if there is already a Puppetfile
@@ -879,16 +879,16 @@ def initialize_project
         end
 
         installer = Bolt::ModuleInstaller.new(outputter, pal)
-        installer.install(options[:modules], puppetfile, moduledir)
+        installer.install(options[:modules], puppetfile, moduledir, config.module_install)
       end
 
       # If either bolt.yaml or bolt-project.yaml exist, the user has already
       # been warned and we can just finish project creation. Otherwise, create a
       # bolt-project.yaml with the project name in it.
-      unless config.exist? || old_config.exist?
+      unless project_file.exist? || config_file.exist?
         begin
           content = { 'name' => name }
-          File.write(config.to_path, content.to_yaml)
+          File.write(project_file.to_path, content.to_yaml)
           outputter.print_message "Successfully created Bolt project at #{project}"
         rescue StandardError => e
           raise Bolt::FileError.new("Could not create bolt-project.yaml at #{project}: #{e.message}", nil)
@@ -900,7 +900,7 @@ def initialize_project
 
     # Installs modules declared in the project configuration file.
     #
-    def install_project_modules(project, force, resolve)
+    def install_project_modules(project, config, force, resolve)
       assert_project_file(project)
 
       unless project.modules
@@ -909,19 +909,21 @@ def install_project_modules(project, force, resolve)
         return 0
       end
 
+      modules   = project.modules || []
       installer = Bolt::ModuleInstaller.new(outputter, pal)
 
-      ok = installer.install(project.modules,
+      ok = installer.install(modules,
                              project.puppetfile,
                              project.managed_moduledir,
+                             config,
                              force: force,
                              resolve: resolve)
       ok ? 0 : 1
     end
 
     # Adds a single module to the project.
     #
-    def add_project_module(name, project)
+    def add_project_module(name, project, config)
       assert_project_file(project)
 
       modules   = project.modules || []
@@ -931,7 +933,8 @@ def add_project_module(name, project)
                          modules,
                          project.puppetfile,
                          project.managed_moduledir,
-                         project.project_file)
+                         project.project_file,
+                         config)
       ok ? 0 : 1
     end
 

diff --git a/lib/bolt/config.rb b/lib/bolt/config.rb
@@ -40,7 +40,6 @@ def self.from_project(project, overrides = {})
                logs << { debug: "Loaded configuration from #{project.config_file}" } if File.exist?(project.config_file)
                c
              end
-
       data = load_defaults(project).push(
         filepath: project.config_file,
         data: conf,
@@ -212,6 +211,7 @@ def initialize(project, config_data, overrides = {})
         'concurrency'         => default_concurrency,
         'format'              => 'human',
         'log'                 => { 'console' => {} },
+        'module-install'      => {},
         'plugin_hooks'        => {},
         'plugins'             => {},
         'puppetdb'            => {},
@@ -334,7 +334,8 @@ def deep_clone
 
       # Filter hashes to only include valid options
       @data['apply_settings'] = @data['apply_settings'].slice(*OPTIONS['apply_settings'][:properties].keys)
-      @data['puppetfile'] = @data['puppetfile'].slice(*OPTIONS['puppetfile'][:properties].keys)
+      @data['puppetfile']     = @data['puppetfile'].slice(*OPTIONS['puppetfile'][:properties].keys)
+      @data['module-install'] = @data['module-install'].slice(*OPTIONS['module-install'][:properties].keys)
     end
 
     private def normalize_log(target)
@@ -528,6 +529,10 @@ def transport
       @data['transport']
     end
 
+    def module_install
+      @project.module_install || @data['module-install']
+    end
+
     # Check if there is a case-insensitive match to the path
     def check_path_case(type, paths)
       return if paths.nil?

diff --git a/lib/bolt/config/options.rb b/lib/bolt/config/options.rb
@@ -227,6 +227,41 @@ module Options
           _example: ["~/.puppetlabs/bolt/modules", "~/.puppetlabs/bolt/site-modules"],
           _default: ["project/modules", "project/site-modules", "project/site"]
         },
+        "module-install" => {
+          description: "Options that configure where Bolt downloads modules from. This option is only used when "\
+                       "installing modules using the `bolt module add|install` commands and "\
+                       "`Add|Install-BoltModule` cmdlets.",
+          type: Hash,
+          properties: {
+            "forge" => {
+              description: "A subsection that can have its own `proxy` setting to set an HTTP proxy for Forge "\
+                           "operations only, and a `baseurl` setting to specify a different Forge host.",
+              type: Hash,
+              properties: {
+                "baseurl" => {
+                  description: "The URL to the Forge host.",
+                  type: String,
+                  format: "uri",
+                  _example: "https://forge.example.com"
+                },
+                "proxy" => {
+                  description: "The HTTP proxy to use for Forge operations.",
+                  type: String,
+                  format: "uri",
+                  _example: "https://my-forge-proxy.com:8080"
+                }
+              },
+              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://my-forge-proxy.com:8080" }
+            },
+            "proxy" => {
+              description: "The HTTP proxy to use for Git and Forge operations.",
+              type: String,
+              format: "uri",
+              _example: "https://my-proxy.com:8080"
+            }
+          },
+          _plugin: false
+        },
         "modules" => {
           description: "A list of module dependencies for the project. Each dependency is a map of data specifying "\
                        "the module to install. To install the project's module dependencies, run the `bolt module "\
@@ -360,7 +395,8 @@ module Options
           _plugin: true
         },
         "puppetfile" => {
-          description: "A map containing options for the `bolt puppetfile install` command.",
+          description: "A map containing options for the `bolt puppetfile install` command and "\
+                       "`Install-BoltPuppetfile` cmdlet.",
           type: Hash,
           properties: {
             "forge" => {
@@ -375,19 +411,19 @@ module Options
                   _example: "https://forge.example.com"
                 },
                 "proxy" => {
-                  description: "The HTTP proxy to use for Git and Forge operations.",
+                  description: "The HTTP proxy to use for Forge operations.",
                   type: String,
                   format: "uri",
-                  _example: "https://forgeapi.example.com"
+                  _example: "https://my-forge-proxy.com:8080"
                 }
               },
-              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://forgeapi.example.com" }
+              _example: { "baseurl" => "https://forge.example.com", "proxy" => "https://my-forge-proxy.com:8080" }
             },
             "proxy" => {
               description: "The HTTP proxy to use for Git and Forge operations.",
               type: String,
               format: "uri",
-              _example: "https://forgeapi.example.com"
+              _example: "https://my-proxy.com:8080"
             }
           },
           _plugin: false
@@ -505,6 +541,7 @@ module Options
         format
         inventory-config
         log
+        module-install
         plugin_hooks
         plugins
         puppetdb
@@ -523,6 +560,7 @@ module Options
         inventoryfile
         log
         modulepath
+        module-install
         modules
         name
         plans