Revamping the download mode file (#1508)

Also doing a few things in the process:

- Deprecating the documentation that references the filter file
- Changing the order of the configuration documentation to put the deprecated documents at the bottom of the config section
- Adding a note to the disk storage documentation about pre-filling the disk cache

Fixes #1501

docs/content/configuration/authentication.md

@@ -1,7 +1,7 @@
 ---
-title: Configuring Authentication
+title: Authentication to private repositories
 description: Configuring Authentication on Athens
-weight: 1
+weight: 2
 ---
 
 ## Authentication

docs/content/configuration/download.md

@@ -1,28 +1,42 @@
 ---
-title: Download Mode
+title: The download mode file
 description: What to do when a module is not in storage
-weight: 2
+weight: 1
 ---
 
-Athens accepts an HCL formatted Download File that has instructions for how it should behave when a module@version isn't found in its storage.
+Athens accepts an [HCL](https://github.com/hashicorp/hcl) formatted file that has instructions for how it should behave when a module@version isn't found in its storage. This functionality gives Athens the flexibility configure Athens to fit your organization's needs. The most popular uses of this download file are:
 
-You configure this download file by setting the `ATHENS_DOWNLOAD_MODE` environment variable in one of two ways:
+- Configure Athens to never download or serve a module or group of modules
+- Redirect to a different module proxy for a module or group of modules
 
-1. Set its value to `file:$FILE_PATH`, where `FILE_PATH` is the path to the HCL file
-1. Set its value to `custom$BASE_64` where `BASE_64` is the base64 encoded HCL file
+This document will outline how to use this file - called the download mode file - to accomplish these tasks and more.
 
-## What should Athens do when a module@version is not found in storage?
+>Please see the "Use cases" section below for more details on how to enable these behaviors and more.
 
-Say a client sends an HTTP request with the path `/github.com/pkg/errors/@v/v0.8.1` and Athens
-does not have this module in storage. Athens will look at the Download File for one of the following Modes:
+## Configuration
 
-1. **`sync`**: Synchronously download the module from VCS via `go mod download`, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior. 
+First, once you've created your download mode file, you tell Athens to use it by setting the `DownloadMode` configuration parameter in the `config.toml` file, or setting the `ATHENS_DOWNLOAD_MODE` environment variable. You can set this configuration value to one of two values to tell Athens to use your file:
+
+1. Set its value to `file:$FILE_PATH`, where `$FILE_PATH` is the path to the HCL file
+2. Set its value to `custom$BASE_64` where `$BASE_64` is the base64 encoded HCL file
+
+>Instead of one of the above two values, you can set this configuration to `sync`, `async`, `none`, `redirect`, or `async_redirect`. If you do, the download mode will be set globally rather than for specific sub-groups of modules. See below for what each of these values mean.
+
+## Download mode keywords
+
+If Athens receives a request for the module `github.com/pkg/errors` at version `v0.8.1`, and it doesn't have that module and version in its storage, it will consult the download mode file for specific instructions on what action to take:
+
+1. **`sync`**: Synchronously download the module from VCS via `go mod download`, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior.
 2. **`async`**: Return a 404 to the client, and asynchronously download and persist the module@version to storage.
-3. **`none`**: Return a 404 and do nothing. 
+3. **`none`**: Return a 404 and do nothing.
 4. **`redirect`**: Redirect to an upstream proxy (such as proxy.golang.org) and do nothing after.
-5. **`async_redirect`**: Redirect to an upstream proxy (such as proxy.golang.org) and asynchronously download and persist the module@version to storage. 
+5. **`async_redirect`**: Redirect to an upstream proxy (such as proxy.golang.org) and asynchronously download and persist the module@version to storage.
 
-Furthermore, the Download File can describe any of the above behavior for different modules and module patterns alike using [path.Match](https://golang.org/pkg/path/#Match). Take a look at the following example:
+Athens expects these keywords to be used in conjunction with module patterns (`github.com/pkg/*`, for example). You combine the keyword and the pattern to specify behavior for a specific group of modules.
+
+>Athens uses the Go [path.Match](https://golang.org/pkg/path/#Match) function to parse module patterns.
+
+Below is an example download mode file.
 
 ```javascript
 downloadURL = "https://proxy.golang.org"
@@ -43,25 +57,46 @@ download "github.com/pkg/*" {
 }
 ```
 
-The first two lines describe the behavior and the destination of all packages: redirect to `https://proxy.golang.org` and asynchronously persist the module to storage. 
+The first two lines describe the _default_ behavior for all modules. This behavior is overridden for select module groups below. In this case, the default behavior is:
 
-The following two blocks describe what to do if the requested module matches the given pattern:
+- Immediatley redirect all requests to `https://proxy.golang.org`
+- In the background, download the module from the version control system (VCS) and store it
 
-Any module that matches "github.com/gomods/*" such as "github.com/gomods/athens", will be synchronously fetched, stored, and returned to the user. 
+The rest of the file contains `download` blocks. These override the default behavior for specific groups of modules.
 
-Any module that matches "golang.org/x/*" such as "golang.org/x/text" will just return a 404. Note that this behavior allows the client to set GOPROXY to multiple comma separated proxies so that the Go command can move into the second argument. 
+The first block specifies that any module matching `github.com/gomods/*` (such as `github.com/gomods/athens`) will be downloaded from GitHub, stored, and then returned to the user.
 
-Any module that matches "github.com/pkg/*" such as "github.com/pkg/errors" will be redirected to https://gocenter.io (and not proxy.golang.org) and will also never persist the module to the Athens storage.
+The second block specifies that any module matching `golang.org/x/*` (such as `golang.org/x/text`) will always return a HTTP 404 response code. This behavior ensures that Athens will _never_ store or serve any module names starting with `golang.org/x`.
 
+If a user has their `GOPROXY` environment variable set with a comma separated list, their `go` command line tool will always try the option next in the list. For example, if a user has their `GOPROXY` environment variable set to `https://athens.azurefd.net,direct`, and then runs `go get golang.org/x/text`, they will still download `golang.org/x/text` to their machine. The module just won't come from Athens.
+
+The last block specifies that any module matching `github.com/pkg/*` (such as `github.com/pkg/errors`) will always redirect the `go` tool to https://gocenter.io. In this case, Athens will never persist the given module to its storage.
 
 ## Use cases
 
-So why would you want to use the Download File to configure the behavior above? Here are a few use cases where it might make sense for you to do so: 
+The download mode file is versatile and allows you to configure Athens in a large variety of different ways. Below are some of the mode common.
+
+## Blocking certain modules
 
-**Limited storage:**
+If you're running Athens to serve a team of Go developers, it might be useful to ensure that the team doesn't use a specific group or groups of modules (for example, because of licensing or security issues).
 
-If you have limited storage, then it might be a good idea to only persist some moduels (such as private ones) and then redirect to a public proxy for everything else. 
+In this case, you would write this in your file:
 
-**Limited resources:**
+```hcl
+download "bad/module/repo/*" {
+    mode = "none"
+}
+```
+
+### Preventing storage overflow
+
+If you are running Athens using a [storage backend](./storage) that has limited space, you may want to prevent Athens from storing certain groups of modules that take up a lot of space. To avoid exhausting Athens storage, while still ensuring that the users of your Athens server still get access to the modules you can't store, you would use a `redirect` directive, as shown below:
+
+```hcl
+download "very/large/*" {
+    mode = "redirect"
+    url = "https://reliable.proxy.com"
+}
+```
 
-If you are running Athens with low memory/cpu, then you can redirect all public modules to proxy.golang.org but asynchronously fetch them so that the client does not timeout. At the same time, you can return a 404 for private modules through the `none` mode and let the client (the Go command) fetch private modules directly through `GOPROXY=<athens-url>,direct` 
+>If you use the `redirect` mode, make sure that you specify a `url` value that points to a reliable proxy.

docs/content/configuration/filter.md

@@ -1,9 +1,11 @@
 ---
-title: Filtering modules
+title: Filtering modules (deprecated)
 description: Configuring modules that are stored on the proxy
-weight: 1
+weight: 6
 ---
 
+>Note: the filter file that this page documents is deprecated. Please instead see ["Filtering with the download mode file"](./download) for updated instructions on how to filter modules in Athens.
+
 The proxy supports the following three use cases
 
 1. Fetches a module directly from the source (upstream proxy)
@@ -90,4 +92,4 @@ The currently supported modifiers are
 * `<1.2.3` will enable all versions lower than 1.2.3 (e.g. 1.2.2, 1.0.0 and 0.58.9)
   * Formally, `x.y.z` where `x <= 1`, `y < = 2` and `z < 3`
 
-This kind of modifiers will work only if a three parts semantic version is specified. For example, `~4.5.6` will work while `~4.5` won't.
+This kind of modifiers will work only if a three parts semantic version is specified. For example, `~4.5.6` will work while `~4.5` won't.

docs/content/configuration/prefill-disk-cache.md

@@ -1,7 +1,7 @@
 ---
-title: Pre-filling the Disk Cache
+title: Pre-filling disk storage
 description: How to pre-fill the disk cache
-weight: 4
+weight: 5
 ---
 
 One of the popular features of Athens is that it can be run completely cut off from the internet. In this case, though, it can't reach out to an upstream (e.g. a VCS or another module proxy) to fetch modules that it doesn't have in storage. So, we need to manually fill up the disk partition that Athens uses with the dependencies that we need.

docs/content/configuration/storage.md

@@ -1,22 +1,32 @@
 ---
 title: Configuring Storage
 description: Configuring Storage in Athens
+weight: 3
 ---
 
 ## Storage
 
 The Athens proxy supports many storage types:
 
-1. [Memory](#memory)
-1. [Disk](#disk)
-1. [Mongo](#mongo)
-1. [Google Cloud Storage](#google-cloud-storage)
-1. [AWS S3](#aws-s3)
-1. [Minio](#minio)
-    1. [DigitalOcean Spaces](#digitalocean-spaces)
-    1. [Alibaba OSS](#alibaba-oss)
-    1. and other S3 / Minio compatible interfaces
-1. [Azure Blob Storage](#azure-blob-storage)
+- [Storage](#storage)
+- [Memory](#memory)
+      - [Configuration:](#configuration)
+- [Disk](#disk)
+      - [Configuration:](#configuration-1)
+- [Mongo](#mongo)
+      - [Configuration:](#configuration-2)
+- [Google Cloud Storage](#google-cloud-storage)
+      - [Configuration:](#configuration-3)
+- [AWS S3](#aws-s3)
+      - [Configuration:](#configuration-4)
+- [Minio](#minio)
+      - [Configuration:](#configuration-5)
+    - [DigitalOcean Spaces](#digitalocean-spaces)
+      - [Configuration:](#configuration-6)
+    - [Alibaba OSS](#alibaba-oss)
+      - [Configuration:](#configuration-7)
+- [Azure Blob Storage](#azure-blob-storage)
+      - [Configuration:](#configuration-8)
 
 All of them can be configured using `config.toml` file. You need to set a valid driver in `StorageType` value or you can set it in environment variable `ATHENS_STORAGE_TYPE` on your server.
 Also for most of the drivers you need to provide additional configuration data which will be described below.
@@ -37,6 +47,8 @@ This storage doesn't need any specific configuration and it's also used by defau
 
 Disk storage allows modules to be stored on a file system. The location on disk where modules will be stored can be configured.
 
+>You can pre-fill disk-based storage to enable Athens deployments that have no access to the internet. See [here](./prefill-disk-cache) for instructions on how to do that.
+
 ##### Configuration:
 
     # StorageType sets the type of storage backend the proxy will use.
@@ -318,4 +330,4 @@ It assumes that you already have the following:
 
             # Name of container in the blob storage
             # Env override: ATHENS_AZURE_CONTAINER_NAME
-            ContainerName = "MY_AZURE_BLOB_CONTAINER_NAME"
+            ContainerName = "MY_AZURE_BLOB_CONTAINER_NAME"

docs/content/configuration/sumdb.md

@@ -1,7 +1,7 @@
 ---
-title: Checksum DB
-description: Proxying A Checksum DB API
-weight: 2
+title: Proxying a checksum database API
+description: How to configure Athens to proxy a checksum database API, and why you might want to.
+weight: 4
 ---
 
 ## Proxying A Checksum DB

docs/content/configuration/upstream.md

@@ -1,18 +1,20 @@
 ---
-title: Configuring Upstream Proxy to use an Upstream Go Modules Repository
+title: Using an upstream Go modules repository (deprecated)
 description: How to Configure Athens to Fetch Missing Modules From an Upstream Module Repository Like GoCenter, or Another Athens Server
-weight: 1
+weight: 7
 ---
 
-By default, Athens fetches module code from an upstream VCS like github.com, but this can be configured to use a Go modules repository like GoCenter or another Athens Server.
+>Note: the filter file that this page documents is deprecated. Please instead see ["Filtering with the download mode file"](./download) for updated instructions on how to set upstream repositories in Athens.
+
+By default, Athens fetches module code from an upstream version control system (VCS) like github.com, but this can be configured to use a Go modules repository like GoCenter or another Athens Server.
 
 1. Create a filter file (e.g ```/usr/local/lib/FilterForGoCenter```) with letter `D` (stands for "direct access") in first line. For more details, please refer to documentation on  - [Filtering Modules](/configuration/filter)
 
     ```
     # FilterFile for fetching modules directly from upstream
     D
     ```
-1. If you are not using a config file, create a new config file (based on the sample config.dev.toml) and edit values to match your environment).
+2. If you are not using a config file, create a new config file (based on the sample config.dev.toml) and edit values to match your environment).
 Additionally in the current or new config file, set the following parameters as suggested:
 
     ```

docs/content/design/_index.md

@@ -1,7 +1,7 @@
 ---
 title: "The Design of Athens"
 date: 2018-09-20T15:37:49-07:00
-weight: 5
+weight: 3
 ---
 
 This section of the documentation details the design of Athens. You can read the code and ask plenty of questions (which we're always happy to answer!), but we want to take some time here to give you a head start by describing how Athens is designed in words and diagrams, rather than code.