🍀 Proposal: Add Remote State Support

[algobot76]

Description

It is easy to use dtm for personal use. However, it's might not be enough for teams: everyone has a copy of state that may or may not be up-to-date and it's hard to manually update the copies to the most recent state as they are stored across different computers. A remote seems like a solution to this problem.

Describe the Proposal

To support remote state, dtm must satisfy the following requirements:

• Remote storage: dtm should be able to store the current state on a server (e.g. AWS S3).

• Locking: dtm should prevent concurrent writes to the remote state, otherwise, data corruption may occur.

• Security: dtm should not store the state in plain text. Encryption is needed to prevent security issues.

• Synchronization: dtm should provide a painless way for team members to get the latest state. Also, any instructions to update an old copy of the state should be not be executed.

Has the Feature been Requested Before?

no

Describe the Alternatives You Have Considered

n/a

Additional Context

n/a

[IronCore864]

A very good proposal!

At the moment, we do not have a whole lot of users yet, so maybe we can leave the state locking part out; an AWS S3 backend would be highly appreciated.

[algobot76]

A very good proposal!

At the moment, we do not have a whole lot of users yet, so maybe we can leave the state locking part out; an AWS S3 backend would be highly appreciated.

S3 backend will be a good starting point :)

[daniel-hutao]

Since the tools we want to manage are mainly open source, why not support a free "remote backend" first? Such as ETCD/MySQL and so on. These backends will also be more readily available for many users. I also believe many users' environment is air-gapped, or whose teams do not have a chance to use commercial cloud storage.

If we implement ETCD backend first, we don't have to deploy ETCD on "remote host". Of course, ETCD can also be deployed on the machine where dtm is located. At this time, ETCD is essentially local, but we can call it remote, because when a user chooses to share an ETCD across teams, it is remote, and dtm does NOT Perceive how far this distance is.

[IronCore864]

I think both s3 and etcd/mysql have their own values.

S3 has a great "pro", which is it's almost a universal protocol, supported by many cloud providers and even open-source databases. The con, of course, is the pricing (although minimum).

etcd is open-source, free to use, but the users will have to deploy it themselves. Similar for mysql.

If we have to choose one, which do you guys prefer?

[daniel-hutao]

I'm not familiar with s3, so my vote may not be neutral enough.
@algobot76 I'd like to leave the decision to you.
S3? ETCD? MySql? Make a decision using your best judgement.

[algobot76]

I'm not familiar with s3, so my vote may not be neutral enough. @algobot76 I'd like to leave the decision to you. S3? ETCD? MySql? Make a decision using your best judgement.

I prefer S3 because:

1. S3 is probably the most popular remote storage for files.
2. ETCD is mainly known by K8S developers
3. Both MySQL & ETCD require deployment by users, which is not very convenient.
4. Also, using S3 means the users don't have to take care of security themselves.