feat: add http datasource #11658

teddylear · 2022-03-14T00:14:42Z

Description

Creating http datasource similar to terraform http data source.

Testing

Commands to run tests:

make 
PACKER_ACC=1 go test -count 1 -v ./datasource/http/data_acc_test.go  -timeout=2m

Closes #11552

teddylear · 2022-03-15T23:47:53Z

@nywilken Looks like this needs Vercel permission, is that something you can please grant so CI can run. If there is something I need to run on my end please let me know. Thanks!

teddylear · 2022-05-04T22:57:15Z

@nywilken @azr Is there something I'm missing to trigger the markdown file builds? Or should I just push another commit to get this firing?

sylviamoss

Thanks for opening the PR! I think is looking good so far, but I miss the documentation. Could add it, please?
The documentation for this would be placed in website/content/docs/datasources, you will also have to add them to the navigation in website/data/docs-nav-data.json.

datasource/http/data.go

sylviamoss · 2022-05-06T08:16:18Z

datasource/http/data.go

+
+// Most of this code comes from http terraform provider data source
+// https://github.com/hashicorp/terraform-provider-http/blob/main/internal/provider/data_source.go
+func (d *Datasource) Execute() (cty.Value, error) {


The returned error message can be improved a little bit to help understand the step it failed.

Added some print statements before throwing errors to make them more traceable. But I assume we still want to throw current error up the chain.

sylviamoss · 2022-05-06T08:17:53Z

website/content/partials/datasource/http/Config-not-required.mdx

@@ -0,0 +1,5 @@
+<!-- Code generated from the comments of the Config struct in datasource/http/data.go; DO NOT EDIT MANUALLY -->


You can use these generated partials in the docs. See the other datasources documentation as an example.

Updated PR with docs

sylviamoss · 2022-05-06T08:18:49Z

datasource/http/data_acc_test.go

+var testDatasource404Url string
+
+func TestHttpDataSource(t *testing.T) {
+	tests := []struct {


Thanks 😄

Co-authored-by: Sylvia Moss <sylviamoss.m@gmail.com>

… http-data-resource

… more traceable

teddylear · 2022-05-07T00:57:30Z

@sylviamoss Updated PR based on feedback. Please let me know if I should update anything else.

website/content/docs/datasources/http/http.md

sylviamoss · 2022-05-10T08:47:33Z

website/content/partials/datasource/http/Config.mdx

@@ -0,0 +1,5 @@
+<!-- Code generated from the comments of the Config struct in datasource/http/data.go; DO NOT EDIT MANUALLY -->


Can't find where this one is coming from. Does the CI fail if you remove it? I think it was generated the first time

Let me try to remove it and see what happens

Looks like website changes passed with removing this.

sylviamoss

Thanks for making the changes! I made some suggestions on the docs.

datasource/http/data.go

Co-authored-by: Sylvia Moss <sylviamoss.m@gmail.com>

teddylear · 2022-05-10T21:34:12Z

Updated docs with suggestions (thanks again for suggestions, writing better docs is something I need to improve on). If there's anything else I should change or if you want to to rebase to clean up commit history please let me know.

teddylear · 2022-05-12T01:05:48Z

@sylviamoss It looks like the vercel deployment failed but it's unclear why, any ideas?

sylviamoss · 2022-05-16T09:15:29Z

website/content/docs/datasources/http/http.md

@@ -0,0 +1,49 @@
+---


The build is failing because the file type should be mdx.

The best place to put this doc is directly under the datasources folder, as datasources/http.mdx because it's a single datasource type.

sylviamoss · 2022-05-16T09:16:40Z

website/data/docs-nav-data.json

@@ -712,6 +712,10 @@
            "title": "Image",
            "path": "datasources/hcp/hcp-packer-image"
          },
+          {


Here, you put the datasource under the HCP datasources. You need to move this one up and update the path to accordingly to the changes I proposed above datasources/http

sylviamoss

Thanks for the patience! I've been focused on other stuff in parallel and sometimes it takes longer to switch context

sylviamoss · 2022-05-16T09:20:33Z

To test your website changes locally, you can do

~ cd website
~ make website/build-local
~ make website/local

It requires docker running.

teddylear · 2022-05-16T21:42:50Z

@sylviamoss Thanks again for all the reviews.I think I addressed issues you mentioned and the container builds and runs locally for me. Please let me know if there's anything else I should change.

sylviamoss

Thanks! 👍🏼 LGTM

azr · 2022-05-18T13:08:52Z

datasource/http/data.go

+	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
+	// TODO: How to make a test case for this?
+	if err != nil {
+		fmt.Println("Error creating http request")
+		return cty.NullVal(cty.EmptyObject), err
+	}


I shouldn't be here, don't tell anyone, but the httptest library should help here: https://pkg.go.dev/net/http/httptest#NewServer, you can make it return actual data without starting a server, I think.

Also, NewRequestWithContext uses a global client, and I'd recommend on having a more local one.

🥷

I can make a PR with this update. Thanks!

github-actions · 2022-06-18T02:32:25Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

teddylear marked this pull request as ready for review March 14, 2022 00:22

teddylear requested a review from a team as a code owner March 14, 2022 00:22

feat: add http datasource

1f5f16e

vercel bot deployed to Preview April 4, 2022 20:10 View deployment

sylviamoss reviewed May 6, 2022

View reviewed changes

datasource/http/data.go Outdated Show resolved Hide resolved

sylviamoss reviewed May 6, 2022

View reviewed changes

teddylear and others added 4 commits May 6, 2022 20:03

Merge branch 'main' into http-data-resource

13c407a

Update datasource/http/data.go

9dfa1c0

Co-authored-by: Sylvia Moss <sylviamoss.m@gmail.com>

Merge branch 'http-data-resource' of github.com:teddylear/packer into…

7c926b7

… http-data-resource

fix: add http datasource docs and add print statements to make errors…

db79495

… more traceable

teddylear requested a review from sylviamoss May 9, 2022 13:45