Terraform backend: Git repository

[dee-kryvenko]

I was entertaining the idea of using an arbitrary Git repository as state backend in Terraform.

I know it sounds like a stupid idea at first, but hear me out!

I explained my idea in details here https://github.com/plumber-cd/terraform-backend-git/blob/master/README.md, there's also an implementation proposal. I wouldn't waste my time trying to implement it as a native backend in terraform without understanding that there's some interest to get it merged among terraform maintainers. But for now, there's fully functional HTTP backend implementation in that repository, that would translate HTTP requests to Git actions.

[CarlosDomingues]

Additionally, the locking support can be implemented using branches - to acquire the lock, the branch with it's name (path) needs to be pushed. If the branch exists - someone else already acquired the lock

Pretty cool!

[dee-kryvenko]

@CarlosDomingues yeah thanks. There's obviously more than one way to implement the locking, but I was aiming to avoid complex git scenarios that would involve merging and potential git conflicts (in other words - to keep local working tree fast-forwardable at all times). Simple branch push would do and it would be an atomic operation. If the push fails (fast-forward push not possible) - someone else already had the lock.

[dee-kryvenko]

Ok I just published my backend implementation here https://github.com/plumber-cd/terraform-backend-git, the README.md has detailed implementation proposal with examples, so I will update issue description now to point to it.

[pkolyvas]

@dee-kryvenko Thanks for submitting this well-described and thought-out issue.

Regarding a new backend, the Terraform Team is currently not merging new remote state backends and we're grateful that you've taken the position of proposing it here first. We have struggled to find maintainers for the backends that are currently shipping with Terraform.

Our current plan is to enable backends as plugins in the similar way as providers are, but we don't yet have a timeline or priority for that work. We're currently focused on our work related to modules and HashiCorp's provider registry: https://github.com/hashicorp/terraform/milestone/9

We very much appreciate you moving ahead with the HTTP backend in the mean time and I will of course leave this open such that when we can create backends as plugins, it's ready to go or be picked back up.

[dee-kryvenko]

@pkolyvas thanks for transparency. Is there any docs on how pluggable backends planned to be implemented and will you accept any contributions towards that work? I am thinking of a a simple grcp backend implementation, probably would be a good start. I understand that you're not accepting new backends atm, but how about grcp backend that will effectively be a first step towards making backends pluggable?

Also I found a couple of issues with http backend and I'm also having some features in mind for it, specifically around security and authentication - will you be willing to accept any contributions in that space?

[binlab]

It would be nice to have it!

[pkolyvas]

@dee-kryvenko Sorry for the delay.

We don't yet have design docs to share. Pluggable backends are something Terraform Core will work on in coordination with the team which now produces the Terraform Plugin SDK: https://github.com/hashicorp/terraform-plugin-sdk

Unfortunately we're pushing towards the next major release of Terraform at the moment and backends are on the back-burner. I will share as much information as I can once we have it.

While your proposal for gRPC backend is an interesting one, one of the major concerns we have is introducing yet another area of Terraform which might be adopted in such a way as to make our plans to convert backends to plugins even more difficult. Once a feature is released it is often complex to walk back/change or modify. There are additional concerns, such as initializing a gRPC backend itself which must, in turn, initialize, be configured with, or configure another piece of software (the backend plugin itself). I hope, even with my musings here (which is all they are) that I've made a case (if marginal one) for not going that route at the moment.

For the HTTP backend - we definitely would appreciate the help. :)

One of the many things we're hoping to encourage is a more open design process for feature enhancements. The idea is that once someone is willing to take up the development work for an enhancement a technical proposal can be discussed with the team before code is committed. This isn't to control contributions, but rather that the team would prefer to help work towards PRs that can be merged more easily rather than incredibly difficult PRs just appearing out of the blue. We've recently begun updating our contribution guide to that effect. Our hope is that we can drive towards a process that encourages contributions rather than dissuades them.

[danieldreier]

@dee-kryvenko I'm going to close this issue because of the reasons @pkolyvas has described - I don't think it makes sense to hold this open now that we have decided not to do this in this codebase, because when a pluggable backend system is available, you won't need to open an issue in this repository to build a new backend.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.