(BOLT-1589) Support installing any git-based module

bolt.gemspec

@@ -57,7 +57,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency "net-ssh-krb", "~> 0.5"
   spec.add_dependency "orchestrator_client", "~> 0.5"
   spec.add_dependency "puppet", ">= 6.18.0"
-  spec.add_dependency "puppetfile-resolver", "~> 0.5"
+  spec.add_dependency "puppetfile-resolver", ">= 0.6.2", "< 1.0"
   spec.add_dependency "puppet-resource_api", ">= 1.8.1"
   spec.add_dependency "puppet-strings", "~> 2.3"
   spec.add_dependency "r10k", "~> 3.10"

documentation/bolt_installing_modules.md

@@ -10,9 +10,9 @@ You can use the command line to:
   downloaded a project from source control and want to install its dependencies.
 - Update your project's modules.
 
-If you need to install a module from a GitHub repository or an alternate Forge,
-or you need to use a Forge proxy, you can manually configure your Bolt modules
-in your Bolt project configuration file (`bolt-project.yaml`).
+If you need to install a module from a git repository or an alternate Forge, or
+you need to use a Forge proxy, you can manually configure your Bolt modules in
+your Bolt project configuration file (`bolt-project.yaml`).
 
 ## Create a Bolt project with pre-installed modules
 
@@ -216,7 +216,7 @@ To specify a git module, use the following keys in the specification:
 
 | Key | Description | Required |
 | --- | --- | :-: |
-| `git` | The URI to the GitHub repository. URI must begin with either `https://github.com`or `git@github.com`. | ✓ |
+| `git` | The URI to the git repository. URI must begin with either `git@`, `http://`, or `https://`. | ✓ |
 | `name` | The name of the module. Bolt uses this name for the module in the Puppetfile, the directory that the module's contents are downloaded to, and as a namespace for the module's content. To avoid errors, make sure this name matches the name specified in the module's `metadata.json`. **Required if `resolve` is `false`.** | |
 | `ref` | The git reference to checkout. Can be either a branch, commit, or tag. | ✓ |
 | `resolve` | Boolean. Whether to resolve the module's dependencies when installing modules. | |
@@ -233,8 +233,6 @@ modules:
     ref: '7.0.0'
 ```
 
-Bolt only supports installing git modules from GitHub.
-
 ## Pin a module version
 
 If you need to pin a module in your Bolt project to a specific version, you can
@@ -364,8 +362,6 @@ resolve dependencies for.
 You might want to skip dependency resolution for a module if:
 
 - The module has outdated or incorrect metadata.
-- The module is a git module hosted in a repository other than a public GitHub
-  repository. For example, a private GitHub repository.
 - Bolt can't cleanly resolve the module's dependencies.
 
 You can configure Bolt to skip dependency resolution for a module by setting the
@@ -378,8 +374,8 @@ dependencies, Bolt generates a Puppetfile with the resolved modules and
 dependencies, as well as the modules it did not resolve dependencies for.
 
 For example, if your project includes the `puppetlabs/ntp` Forge module and a
-git module named `private_module` hosted in a private GitHub repository, you can
-configure Bolt to skip dependency resolution for `private_module`:
+git module named `private_module` that has incorrect metadata, you can configure
+Bolt to skip dependency resolution for `private_module`:
 
 ```yaml
 # bolt-project.yaml
@@ -453,11 +449,6 @@ without resolving dependencies. The process for manually managing your modules
 uses the new `module` subcommand, and replaces the now deprecated `puppetfile`
 subcommand.
 
-The most common scenario where Bolt can't resolve module dependencies is when
-a project includes git modules that are in a repository other than a public
-GitHub repository. If your project includes this type of module, you must
-manually manage your project's Puppetfile.
-
 To manually manage a project's Puppetfile and install modules without resolving
 dependencies, follow these steps:
 
@@ -466,10 +457,9 @@ dependencies, follow these steps:
    the Puppetfile. For example:
 
    ```ruby
-   # Modules from a private git repository
-   mod 'private-module', git: 'https://github.com/bolt-user/private-module.git', ref: 'main'
-
-   # Modules from an alternate Forge
+   mod 'private-module',
+    git: 'https://github.com/bolt-user/private-module.git',
+    ref: 'main'
    mod 'puppetlabs/apache', '5.7.0'
    mod 'puppetlabs/stdlib', '6.5.0'
    mod 'puppetlabs/concat', '6.3.0'
@@ -505,9 +495,7 @@ project](projects.md#migrate-a-bolt-project).
 In some cases, Bolt is unable to resolve module dependencies and manage your
 project's modules for you. If Bolt can't resolve your module dependencies, you
 can manage your project's Puppetfile manually and use Bolt to install the
-modules listed in the Puppetfile without resolving dependencies. The most common
-scenario where Bolt can't resolve module dependencies is when a project includes
-git modules that are in a repository other than a public GitHub repository.
+modules listed in the Puppetfile without resolving dependencies.
 
 The module management feature makes changes to configuration files and changes
 the directory where modules are installed. To migrate your project, do the

lib/bolt/module_installer.rb

@@ -18,7 +18,7 @@ def initialize(outputter, pal)
     # Adds a single module to the project.
     #
     def add(name, specs, puppetfile_path, moduledir, project_file, config)
-      project_specs = Specs.new(specs)
+      project_specs = Specs.new(specs, config)
 
       # Exit early if project config already includes a spec with this name.
       if project_specs.include?(name)
@@ -151,7 +151,7 @@ def install(specs, path, moduledir, config = {}, force: false, resolve: true)
       @outputter.print_message("Installing project modules\n\n")
 
       if resolve != false
-        specs = Specs.new(specs)
+        specs = Specs.new(specs, config)
 
         # If forcibly installing or if there is no Puppetfile, resolve
         # and write a Puppetfile.

lib/bolt/module_installer/resolver.rb

@@ -49,7 +49,7 @@ def resolve(specs, config = {})
             spec_searcher_configuration: spec_searcher_config(config)
           )
         rescue StandardError => e
-          raise Bolt::Error.new(e.message, 'bolt/module-resolver-error')
+          raise Bolt::Error.new("Unable to resolve modules: #{e.message}", 'bolt/module-resolver-error')
         end
 
         # Create the Puppetfile object.

lib/bolt/module_installer/specs.rb

@@ -7,8 +7,10 @@
 module Bolt
   class ModuleInstaller
     class Specs
-      def initialize(specs = [])
-        @specs = []
+      def initialize(specs = [], config = {})
+        @specs  = []
+        @config = config
+
         add_specs(specs)
         assert_unique_names
       end
@@ -49,7 +51,7 @@ def add_specs(*specs)
       #
       private def spec_from_hash(hash)
         return ForgeSpec.new(hash) if ForgeSpec.implements?(hash)
-        return GitSpec.new(hash)   if GitSpec.implements?(hash)
+        return GitSpec.new(hash, @config) if GitSpec.implements?(hash)
 
         raise Bolt::ValidationError, <<~MESSAGE.chomp
           Invalid module specification:

lib/bolt/module_installer/specs/git_spec.rb

@@ -1,9 +1,15 @@
 # frozen_string_literal: true
 
 require 'json'
+require 'net/http'
+require 'open3'
 require 'set'
 
 require_relative '../../../bolt/error'
+require_relative '../../../bolt/logger'
+require_relative '../../../bolt/module_installer/specs/id/gitclone'
+require_relative '../../../bolt/module_installer/specs/id/github'
+require_relative '../../../bolt/module_installer/specs/id/gitlab'
 
 # This class represents a Git module specification.
 #
@@ -17,22 +23,29 @@ class GitSpec
 
         attr_reader :git, :ref, :resolve, :type
 
-        def initialize(init_hash)
-          @resolve    = init_hash.key?('resolve') ? init_hash['resolve'] : true
-          @name       = parse_name(init_hash['name'])
-          @git, @repo = parse_git(init_hash['git'])
-          @ref        = init_hash['ref']
-          @type       = :git
+        def initialize(init_hash, config = {})
+          @logger  = Bolt::Logger.logger(self)
+          @resolve = init_hash.key?('resolve') ? init_hash['resolve'] : true
+          @git     = init_hash['git']
+          @ref     = init_hash['ref']
+          @name    = parse_name(init_hash['name'])
+          @proxy   = config.dig('proxy')
+          @type    = :git
+
+          unless @resolve == true || @resolve == false
+            raise Bolt::ValidationError,
+                  "Option 'resolve' for module spec #{@git} must be a Boolean"
+          end
 
           if @name.nil? && @resolve == false
             raise Bolt::ValidationError,
                   "Missing name for Git module specification: #{@git}. Git module specifications "\
                   "must include a 'name' key when 'resolve' is false."
           end
 
-          unless @resolve == true || @resolve == false
+          unless valid_url?(@git)
             raise Bolt::ValidationError,
-                  "Option 'resolve' for module spec #{@git} must be a Boolean"
+                  "Invalid URI #{@git}. Valid URIs must begin with 'git@', 'http://', or 'https://'."
           end
         end
 
@@ -57,22 +70,6 @@ def self.implements?(hash)
           match[:name]
         end
 
-        # Gets the repo from the git URL.
-        #
-        private def parse_git(git)
-          return [git, nil] unless @resolve
-
-          repo = if git.start_with?('git@github.com:')
-                   git.split('git@github.com:').last.split('.git').first
-                 elsif git.start_with?('https://github.com')
-                   git.split('https://github.com/').last.split('.git').first
-                 else
-                   raise Bolt::ValidationError, invalid_git_msg(git)
-                 end
-
-          [git, repo]
-        end
-
         # Returns true if the specification is satisfied by the module.
         #
         def satisfied_by?(mod)
@@ -88,14 +85,6 @@ def to_hash
           }
         end
 
-        # Returns an error message that the provided repo is not a git repo or
-        # is private.
-        #
-        private def invalid_git_msg(repo_name)
-          "#{repo_name} is not a public GitHub repository. See https://pup.pt/no-resolve "\
-            "for information on how to install this module."
-        end
-
         # Returns a PuppetfileResolver::Model::GitModule object for resolving.
         #
         def to_resolver_module
@@ -107,90 +96,53 @@ def to_resolver_module
           end
         end
 
-        # Resolves the module's title from the module metadata. This is lazily
-        # resolved since Bolt does not always need to know a Git module's name.
+        # Returns the module's name.
         #
         def name
-          @name ||= begin
-            url      = "https://github.com/#{@repo}/#{sha}/metadata.json"
-            response = make_request(:Get, url)
-
-            case response
-            when Net::HTTPOK
-              body = JSON.parse(response.body)
-
-              unless body.key?('name')
-                raise Bolt::Error.new(
-                  "Missing name in metadata.json at #{git}. This is not a valid module.",
-                  "bolt/missing-module-name-error"
-                )
-              end
-
-              parse_name(body['name'])
-            else
-              raise Bolt::Error.new(
-                "Missing metadata.json at #{git}. This is not a valid module.",
-                "bolt/missing-module-metadata-error"
-              )
-            end
-          end
+          @name ||= parse_name(id.name)
         end
 
-        # Resolves the SHA for the specified ref. This is lazily resolved since
-        # Bolt does not always need to know a Git module's SHA.
+        # Returns the SHA for the module's ref.
         #
         def sha
-          @sha ||= begin
-            url      = "https://api.github.com/repos/#{@repo}/commits/#{ref}"
-            headers  = ENV['GITHUB_TOKEN'] ? { "Authorization" => "token #{ENV['GITHUB_TOKEN']}" } : {}
-            response = make_request(:Get, url, headers)
-
-            case response
-            when Net::HTTPOK
-              body = JSON.parse(response.body)
-              body['sha']
-            when Net::HTTPUnauthorized
-              raise Bolt::Error.new(
-                "Invalid token at GITHUB_TOKEN, unable to resolve git modules.",
-                "bolt/invalid-git-token-error"
-              )
-            when Net::HTTPForbidden
-              message = "GitHub API rate limit exceeded, unable to resolve git modules. "
-
-              unless ENV['GITHUB_TOKEN']
-                message += "To increase your rate limit, set the GITHUB_TOKEN environment "\
-                          "variable with a GitHub personal access token."
-              end
-
-              raise Bolt::Error.new(message, 'bolt/github-api-rate-limit-error')
-            when Net::HTTPNotFound
-              raise Bolt::Error.new(invalid_git_msg(git), "bolt/missing-git-repository-error")
-            else
+          id.sha
+        end
+
+        # Gets the ID for the module based on the specified ref and git URL.
+        # This is lazily resolved since Bolt does not always need this information,
+        # and requesting it is expensive.
+        #
+        private def id
+          @id ||= begin
+            # The request methods here return an ID object if the module name and SHA
+            # were found and nil otherwise. This lets Bolt try multiple methods for
+            # finding the module name and SHA, and short circuiting as soon as it does.
+            module_id = Bolt::ModuleInstaller::Specs::ID::GitHub.request(@git, @ref, @proxy) ||
+                        Bolt::ModuleInstaller::Specs::ID::GitLab.request(@git, @ref, @proxy) ||
+                        Bolt::ModuleInstaller::Specs::ID::GitClone.request(@git, @ref, @proxy)
+
+            unless module_id
               raise Bolt::Error.new(
-                "Ref #{ref} at #{git} is not a commit, tag, or branch.",
-                "bolt/invalid-git-ref-error"
+                "Unable to locate metadata and calculate SHA for ref #{@ref} at #{@git}. This may "\
+                "not be a valid module. For more information about how Bolt attempted to locate "\
+                "this information, check the debugging logs.",
+                'bolt/missing-module-metadata-error'
               )
             end
+
+            module_id
           end
         end
 
-        # Makes a generic HTTP request.
+        # Returns true if the URL is valid.
         #
-        private def make_request(verb, url, headers = {})
-          require 'net/http'
-
-          uri      = URI.parse(url)
-          opts     = { use_ssl: uri.scheme == 'https' }
+        private def valid_url?(url)
+          return true if url.start_with?('git@')
 
-          Net::HTTP.start(uri.host, uri.port, opts) do |client|
-            request = Net::HTTP.const_get(verb).new(uri, headers)
-            client.request(request)
-          end
-        rescue StandardError => e
-          raise Bolt::Error.new(
-            "Failed to connect to #{uri}: #{e.message}",
-            "bolt/http-connect-error"
-          )
+          uri = URI.parse(url)
+          uri.is_a?(URI::HTTP) && uri.host
+        rescue URI::InvalidURIError
+          false
         end
       end
     end