Introduce additional folder layer under org folder

[seymen]

Problem: In a multiple-org scenario (e.g. non-prod and prod orgs), it is not possible to manage org level entities separately per org.

Current folder structure:

org
|_ apiproducts.json

Suggestion is to introduce an additional folder layer under org:

org
|_ foo-non-prod
        |_ apiproducts.json
|_ foo-prod
        |_ apiproducts.json

[ssvaidyanathan]

@msadasivam - This will have impact to the structure for org under resources, should we release this as 1.3 ?

[msadasivam]

The plugin doesn't assume "resources/edge" as a default for config.dir.

This was the original thinking. Will this work?

non-prod
-Dapigee.config.dir=resources/myenterprise-nonprod
prod
-Dapigee.config.dir=resources/myenterprise

[ssvaidyanathan]

Then this is creating duplicate directory structure with all the nested directories

[msadasivam]

The git structure would reflect the org structure.
resources/myenterprise/prod
resources/myenterprise/staging
resources/myenterprise-nonprod/dev
resources/myenterprise-nonprod/uat
resources/myenterprise-nonprod/test

This structure will hold the config for each env organized by orgs. The APIs will not live in this structure .

[ssvaidyanathan]

Then in that case, I cannot use the same maven command to install org and env, api configurations. I will have to run it separately and also maintain separate directories

[msadasivam]

Yes, APIs and config will be 2 different mvn commands. The "Further organization for even larger config files" section in this page has this structure documented.
https://github.com/apigee/apigee-config-maven-plugin/wiki/Organize-config-in-a-large-project
Expect this to be already in a fairly complex situation.

[ssvaidyanathan]

should we go ahead with this change or go ahead with the wiki link provided ?

[gsjurseth]

Why not have something like an org-overrides-folder property wherein you drop those files that override anything that may otherwise be picked up... So for apiProducts you could have:

config/org/apiProducts.json

For the default but then

org-overrides/myFancyShmancyOrg/apiProducts.json

For an override if we're working on the org named: myFancyShmancyOrg.

This way we don't have to duplicate the entire structure and create a place where we're managing the same data more than once.

[msadasivam]

@gsjurseth interesting idea, can we use this one. This is probably worth a separate thread. #50

[msadasivam]

@seymen what has been your experience so far on the current design and alternative ? do we have enough to attempt a revamp.

[karsten-wagner]

All, I am not sure if this is still open or closed by one of the contributions since this issue opened...
I feel like we are successfully using a multi-file config setup that covers all organizations and all environments we have.

The folder structure typically looks like this:

repository-root
├── proxy-1                             <-- This allows multiple proxies/sharedflows in a single repository when needed
│   ├── apiproxy                        <-- This is the part for the apigee-edge-maven-plugin
│   └── config                          <-- This is the root folder for all configuration
│       ├── org-1                       <-- Ideally name the same way as your organization technical name
│       │   ├── org
│       │   │   ├── apiproducts.json
│       │   │   ├── developers.json
│       │   │   └── apps.json
│       │   └── env
│       │       ├── env-1               <-- Ideally name the same way as your environment technical name
│       │       │   └── kvms.json
│       │       └── env-2
│       │           └── kvms.json
│       └── org-2
│           ├── org
│           │   └── apiproducts.json
│           └── env
│               └── env-3
│                   └── kvms.json
└── proxy-2
    └── ...

Our shared POM covers the case like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    ...
    <build>
        <plugins>
            <plugin><artifactId>maven-resources-plugin</artifactId> ... <!-- Plugin config --></plugin>
            <plugin><artifactId>apigee-edge-maven-plugin</artifactId> ... <!-- Plugin config as per examples in this repo --></plugin>
            <plugin><artifactId>apigee-config-maven-plugin</artifactId> ... <!-- Plugin config as per examples in this repo --></plugin>
        </plugins>
    </build>
    <!-- Environment specific properties -->
    <profiles>
        <profile>
            <id>env-1</id> <!-- We have defined [profile id = environment name] to ease up command following below -->
            <properties>
                <apigee.org>org-1</apigee.org> <!-- org-1 from above folder structure -->
                <apigee.env>env-1</apigee.env> <!-- env-1 from above folder structure -->
                <apigee.profile>${apigee.env}</apigee.profile> <!-- Not clear where this comes from, but is included in all examples - works well like this... -->
                <apigee.config.options>none</apigee.config.options> <!-- Default no-op setting, usually overwritten as command line parameter -->
                <apigee.config.dir>${project.basedir}/config/${apigee.org}</apigee.config.dir> <!-- Automatically derived from above described folder structure by naming convention, can be overwritten as command line parameter -->
                ... <!-- Some other parameters for the deploy plugin and OAuth based token generation -->
            </properties>
        </profile>
        ... <!-- Other environments, usually copy & paste from above profile with only different org/env parameters -->
    </profiles>
</project>

To deploy we execute below command (a shell script with some auto-replacement parameters):

mvn clean install -P env-1 \
  -D "apigee.config.options=create" -D "username=$(git config user.email)" -D "bearer=$(get_token)"

The first line is the bare minimum part of the command, the second line shows how to overwrite some of the parameters defined in the shared POM on the command line.
For convenience we are reading user name from git config and use the "get_token" script to login - this can be replaced by username/password, but with above way the shell script is reusable and we do not have passwords in shell history...

As a final hint - we are uploading the shared POM to our companies Maven repository/artifactory and use it as a dependency in the local proxie's POM.

[ssvaidyanathan]

Thanks @karsten-wagner - Yes, that's been our general recommendation as well. The issue is still opened as there were some conversations/discussion on what to do. But in general, what you suggested is what I have recommended to others as well.

Just curious - you have multiple proxies in the same repo. If only proxy2 was modified, how is your pipeline tracking that deploying that alone? That has been challenging (looking at the git log and then triggering the build), so wanted to check with you. The easiest option is to deploy all the proxies in that repo but thats not a scalable approach.

[karsten-wagner]

@ssvaidyanathan, you are right, during development stage we deploy multiple proxies on every commit. Having multiple in the same repository is usually the case when we have a very close link between them.
As an example we have a shared flow with a re-usable feature and a second shared flow which does the error handling for the feature. Usually there is a strong indicator both need to change at the same time.

Once we tag the changes as a ready "release", we are packaging the repository as an artifact to allow later roll back to a previous version if needed. That package is then "deployed" to the target organization/environment via a separate job in CI/CD. During that second phase we can select to deploy only code changes of parts of that package (e.g. single proxy/flow, or just overwrite the products if desired).

[ssvaidyanathan]

Sounds good.. Hence I recommend one repo per proxy.. so you get the flexibility.. Thanks for getting back...