Test Proxy Migration

Test Proxy Server

It is common practice in many of our SDKs to test service client code by recording HTTP requests and responses during a test run against a live endpoint and then playing back the matching responses to requests in subsequent runs.

The basic idea is to have a test server that sits between the client being tested and the live endpoint. Instead of mocking out the communication with the server, the communication can be redirected to a test server.

Major benefits

1. Repo size - Our repos are getting big and the biggest contributor to this issue are recordings. Consolidating on a single solution and recording format makes it easier to have the storage for recordings moved outside of the main repo.
2. Share Code - reusing/centralizing the general code for recording and playback. eg: test server is hard-wired for sanitization/redirection defaults
3. Performance testing - eliminate server-side bottlenecks from a benchmark so instead of contacting the live service, a test service can respond with recorded/cached responses.

Steps:

The roll-out strategy can be split into below phases:

1. Record test recordings with the test-proxy integration
2. Migrate updated recordings to assets repo
3. Using test proxy going forward

1) Record test recordings with the test-proxy integration

Each SDK needs to re-record its test recordings using the test-proxy integration to ensure a consolidated recording format with serialized/sanitized requests and their matching responses.

Steps:

1. Delete old recordings from under src/test/resources/session-records. (In theory it will overwrite them but it's a good idea to just be explicit.)

2. To use the proxy, test classes should extend from TestProxyTestBase instead of TestBase

   public abstract class DocumentAnalysisClientTestBase extends TestProxyTestBase {}

3. Run tests in Record mode using the test-proxy integration.

4. Sanitize secrets:

   Default sanitizers, similar to the use of the RecordingRedactor is registered in the TestProxyUtils.

   Custom sanitizers can be registered using TestProxySanitizer for addressing specific service needs.

   For example, registering a custom sanitizer for redacting the value of json key modelId from the response body looks like the following:

   @Override
   protected void beforeTest() {
       List<TestProxySanitizer> customSanitizer = new ArrayList<>();
       // sanitize value for key: "modelId" in response json body
       customSanitizer.add(new TestProxySanitizer("$..modelId", REPLACEMENT_TEXT, TestProxySanitizerType.BODY_KEY));
       // add sanitizer to Test Proxy Policy
       interceptorManager.addRecordSanitizers(customSanitizer);
   }

Potential gotchas!

Symmetry

The test proxy is much more exact about matching recordings than the old recorder. You may get errors or strange behavior when you try to playback recording, such as output like this:

Header differences:
    <Ocp-Apim-Subscription-Key> is absent in record, value <REDACTED>
    <Authorization> is absent in request, value <Sanitized>

This indicates that there is something different about your test in playback and record modes. Look for places where you're doing something conditional and ensure it is doing the same thing in both modes. If you think that your test is doing the same thing and you're still hitting problems, please reach out.

New pattern for getting the HttpClient

Previously, in lots of tests you might have seen something like this:

SomeClient builder = new SomeClientBuilder()
    // httpClient being an injected parameter, etc.
    .httpClient(httpClient == null ? interceptorManager.getPlaybackClient() : httpClient)
    .build();

This worked because in PLAYBACK mode, TestBase.getHttpClients() would return null. In the new path we want to use the same HttpClient for record and playback. This means getHttpClients() now always returns a client. Thus, this code should change to look like this:

SomeClient builder = new SomeClientBuilder()
    // httpClient being an injected parameter, etc.
    .httpClient(interceptorManager.isPlaybackMode() ? interceptorManager.getPlaybackClient() : httpClient)
    .build();

Adding record polices where they weren't needed

Consider:

SomeClientBuilder builder = new SomeClientBuilder()
    .httpClient(interceptorManager.isPlaybackMode() ? interceptorManager.getPlaybackClient() : httpClient);
if (getTestMode() != TestMode.PLAYBACK) {
    builder.addPolicy(interceptorManager.getRecordPolicy());
}
return builder.buildClient();

Previously, because RecordNetworkCallPolicy had a check for RECORD mode this turned out to be a no-op in LIVE mode. Arguably this is a bug, and is prohibited in the test proxy path. Simply change this to only happen in RECORD:

SomeClientBuilder builder = new SomeClientBuilder()
    .httpClient(interceptorManager.isPlaybackMode() ? interceptorManager.getPlaybackClient() : httpClient);
if (getTestMode() == TestMode.RECORD) {
    builder.addPolicy(interceptorManager.getRecordPolicy());
}
return builder.buildClient();

Use of @BeforeEach

Override beforeTest() instead of overriding BeforeEach to perform any set-up before each test case. Any initialization that occurs in TestBase occurs first before this. Overriding the BeforeEach in test classes with Junit5 will result in it not being inherited.

2) Migrate updated recordings to assets repo

Migrating the test recordings to the asset repo will enable the test proxy to work against repositories and will not require them to emplace their test recordings directly alongside their test implementations.

Steps:

1. Create an asset.json file for each SDK i.e sdk/tables/azure-data-tables/assets.json. The asset.json file basically will allow the test-proxy to restore a set of recordings to a path, then load the recording from that newly gathered data.

Prerequisites:

• The script generate-assets-json.ps1 will execute the initial migration of your recordings from within a language repo to the assets repo as well as creating the assets.json file for those assets.

• Test-proxy needs to be on the machine and in the path. Instructions for that are here.

Running the script:

The script needs to be executed inside an sdk/ or deeper and from within an up-to-date language repository. A good rule here would be look at where the ci.yml is for a service directory. In the case where each library for a given service directory has its own pipelines, at the sdk// level, it is recommended that the assets.json is created there. If the ci.yml exists deeper than the sdk// level, then it is recommended to run the script from that directory.

C:/repo/sdk-for-java/sdk/formrecognizer/azure-ai-formrecognizer> ..\..\..\eng\common\testproxy\transition-scripts\generate-assets-json.ps1 -InitialPush

After running a script, executing a git status from within the language repo, where the script was invoked from, will reflect two primary results:

• A new assets.json will be present in the directory from which they invoked the transition script.
• A bunch of deleted files from where their recordings were before they were pushed to the assets repo.

Running the script without the -InitialPush option will just create the assets.json with an empty tag. No data movement.

3) Using test proxy going forward

After moving recordings to the asset repo, live and playback testing will be the same as it was in the past.

Running tests in Playback mode

1. When running tests in Playback mode, the test-proxy automatically checks out the appropriate tag in each local assets repo and performs testing.
   Here is the result of running tests in playback-mode for a couple of packages within the Java repo:

   

2. Take a look at .breadcrumb file to find which folder contains the recording files for your particular SDK.

   

Running tests in Record mode

1. After running tests in record mode, the newly updated recordings no longer be in the azure-sdk-for-java repo. These updates will be reflected in a git-excluded .assets folder at the root of the repo.
2. You can cd into the folder containing your package's recordings and use git status to view the recording updates. Verify the updates, and use the following command to push these recordings to the azure-sdk-assets repo:

C:/repo/sdk-for-java/>test-proxy push -a sdk/tables/azure-data-tables/assets.json

or running the command providing the context directory

test-proxy push -a C:/repo/sdk-for-python/sdk/tables/azure-data-tables/assets.json

How to set up and use the proxy can be found here.

3. After pushing your recordings, the assets.json file for your package will be updated to point to a new Tag that contains the updates. Include this assets.json update in any pull request to update the recordings pointer in the upstream repo.

More details on the asset-sync feature

Test Proxy CLI commands.

The test proxy enables CLI interactions with the external assets repository using the sdk/<sdk-name>/asset.json file.

Restore operation

To locally clone or pull assets/test-recording associated with a particular asset.json file.

test-proxy restore --assets-json-path <assetsJsonPath>

Reset operation

To reset the local copy of the files back to the version targeted in the given assets.json file.

test-proxy reset --assets-json-path <assetsJsonPath>

Push operation

To push the updated/local assets to the assets repo.

test-proxy push --assets-json-path <assetsJsonPath>

Test Proxy local assets clone repo structure

Test-Proxy maintains a separate clone for each assets.json. The recording files will be located under your repo root under the .assets folder.

    +-------------------------------+
    |  azure-sdk-for-java/        |
    |    sdk/                       |
    |      storage/                 |
    | +------assets.json            |
    | |    appconfiguration/        |
    | | +----assets.json            |
    | | |  keyvault/                |
    | | |    azure-keyvault-secrets |
    | | |      assets.json-------+  |
    | | |    azure-keyvault-keys |  |
    | | |      assets.json---+   |  |
    | | |                    |   |  |
    | | |.assets/            |   |  |
    | | +--AuN9me8zrT/       |   |  |
    | |      <sparse clone>  |   |  |
    | +----5hgHKwvMaN/       |   |  |
    |        <sparse clone>  |   |  |
    |      AuN9me8zrT--------+   |  |
    |        <sparse clone>      |  |
    |      BSdGcyN2XL------------+  |
    |        <sparse clone>         |
    +-------------------------------+
   

For more details/doubts on test-proxy workings, follow Teams channel