Marketplacer Example Webhook Endpoint with .NET

This is a simple example of a webhook endpoint that can be used to consume webhooks generated by Marketplacer. It can be used a starting point for either Operators or Sellers looking to use webhooks as part of their integration.

What you'll learn

This app & associated tutorial is intended to help users new to Marketplacer webhooks to:

• Understand how to set up webhooks in Marketplacer
  • This includes defining the query used to shape the webhook payload
• Use a HTTP Post endpoint (in this case a .NET webapi) to receive webhooks
  • This includes interrogating the HTTP Header metadata to further understand concepts including but not limited to: sequencing, authentication, HMAC etc.

About the app

The app is written in C# using .NET 8.0. it is intended to be a lightweight demonstration of the core concepts a user should be aware of when consuming webhooks from Marketplacer.

Caution

This project is not intended to be a production-ready solution, nor is it covered by Marketplacer support - for more information on the full terms of use, please refer to the license.

Some points to note about this app:

• The app provides a number of different HTTP Post endpoints that you can test and work with, each of these endpoints demonstrate a different webhook concept - the endpoints are as follows:

  • /webhook/test : Basic endpoint that just prints webhook headers to console and responds with a HTTP 200 OK
  • /webhook/auth : Endpoint that checks webhook headers for an auth key
  • /webhook/errors : Endpoint that will respond to the webhook request with a desired HTTP error code, (e.g. 401 Unauthorized).
  • /webhook/hmac : Endpoint that checks for a HMAC header and validates the payload
  • /webhook/sequencing : Endpoint that extracts the sequence header and stores event for backend processing
  • /webhook/payload-parse : Endpoint that takes the webhooks payload and parses the JSON
  • /webhook/payload-diff : Endpoint that takes the webhook payload and compares it to the previous one (to understand what has changed)

• We only cover running the app locally on your development machine, so to that end we will also use Ngrok to expose the endpoint to the internet, (Marketplacer cannot send webhook traffic to a local host app). Ngrok is a 3rd party app used at your own discretion. You should check with your organizations IT and Security teams as to whether its use is permitted within your network.

Important

Marketplacer accepts no responsibility or liability of any kind for the use of Ngrok. Please refer to the software license for more information.

Quick Start

To get up and running with this sample app, here are the steps you should follow:

Download & set up ngrok

Ngrok (and similar apps such as Localtunnel), provide you with a routable url that maps down to a localhost endpoint on your dev machine. This means that you can provide the routable url to the system generating webhooks, (in this case Marketplacer), and those webhooks will eventually make their way down to your local machine, and more specifically this app.

Important

Please refer again to the considerations around the use of ngrok in the About the app section above.

The Ngrok documentation is clear and straightforward and kept up to date by the team at Ngrok, so it should always be your point of reference for setting it up, however the steps we suggest you follow are:

1. Sign up for an Ngrok account and download the app (you should also set up multi-factor authentication on your account).
2. [Optional] Claim your free persistent endpoint. This means that you can use the same endpoint url every time you use Ngrok ,otherwise you’ll get a newly generated endpoint url when Ngrok starts - which is not always ideal.
3. Configure your authentication token. This token is provided to you under your Ngrok account settings along with the command prompt that you need to issue in order to add it to your local Ngrok config file:

ngrok config add-authtoken <your auth token>

That's all for initial set up, we'll come back to running Ngrok a bit later in the guide.

Download .NET SDK

1. Download and install the .NET SDK - this is the runtime environment for our app.

To check that you have installed this correctly , type dotnet --version at the command line, you should get a response similar to the following:

8.0.201

Tip

This app is build with version 8 of the .NET SDK so we recommend that is the version you use.

2. Clone this repo to your local machine:

git clone https://github.com/marketplacer/webhook-endpoint-dotnet.git

3. Run the database migrations

We use a Sqlite database as our persistent data store, so we need to create the database schema that will be used by the app. To do so, at a command prompt change in to the project directory (webhook-endpoint-dotnet) and execute the following:

dotnet ef database update

This should result in the creation of a .db file called WebhookEndpoint.db in the project directory.

Tip

If you get an error along the likes of the ef command being unrecognized you may need to install those tools, to do so at a command prompt type: dotnet tool install --global dotnet-ef

If you want to view the contents of the database, there are a number of tools you can use to do that, the authors personal favorite is DBeaver, a free, exceedingly well featured database management tool that can be used to work with a number of databases, including Sqlite.

4. Start the app

Ensuring that you are still "in" the project directory (webhook-endpoint-dotnet) type the following to run the app:

dotnet run

You should see output similar to the following:

Building...
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: https://localhost:7067
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5205
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: <REDACTED>

Make a note of the https endpoint the app is waiting for requests on, in the above example this is: https://localhost:7067 (be sure to include the port number).

Tip

We are using a self-signed certificate to allow for a "trusted" local https endpoint. You may encounter issues related to this if you have not created and trusted a local development certificate. In order to do that, type: dotnet dev-certs https --trust . You can read more about development certificates here.

Test our endpoint directly

Before attempting to set up webhooks in Marketplacer and proxy them through Ngrok, it's probably a good idea just to test our endpoint directly by making a HTTP POST request to it. We can use the /webhook/test endpoint to achieve this, so the full route we want to hit is:

https://localhost:7067/webhook/test

You can make a post request with whatever tool you are most comfortable with e.g. Postman, Insomnia etc, in this case we are just going to use the command prompt tool curl.

Note

Depending on your OS, you may need to install curl.

So keeping it simple we're just going to send a HTTP POST request as follows:

curl -XPOST -H "Content-type: application/json" 'https://localhost:7067/webhook/test'

If successful you should get a HTTP 200 OK response.

Additionally, if you look at the console output of your running app you should also see the headers attached to the request, something similar to:

Accept: */*
Host: localhost:7067
User-Agent: curl/8.4.0
Content-Type: application/json

Note

What does this mean?: This just means our app is responding to HTTP POST request correctly, and that any other issues we may have are likely to reside elsewhere.

Run and test Ngrok

We now want to run up Ngrok and test that it can proxy requests through to our app. You can run Ngrok in a number of ways ranging from the most basic:

ngrok http 5205

This will:

• Start Ngrok with a dynamic (i.e. changing) url
• Forward http (not https) traffic to a localhost endpoint on port 5205

Note

Looking at the start up logs for our webhook endpoint above, we had an endpoint for http (5205) and an endpoint for https (7067).

This does not satisfy my use-case as I want:

• A static url endpoint
• To use https

To do this, we need to issue the following command to start Ngrok:

ngrok http --domain="my-persistent-domain" https://localhost:7067

If you want to follow along with this method, you will need to substitute in your persistent domain (this can be found in your Ngrok account settings if you have one).

Once successfully started you will see something similar to the following:

To test that traffic makes its way from Ngrok to our webhook endpoint we will make a http post request to our persistent Ngrok endpoint. Again I'm going to use curl to achieve that (ensure that your webhook endpoint is also running):

curl -XPOST -H "Content-type: application/json" 'https://my-persistent-endpoint/webhook/test'

Tip

Don't forget to append /webhook/test to your Ngrok endpoint.

If all was successful we should see a 200 Ok status in the running Ngrok terminal:

Response headers should be printed out as before:

Accept: */*
Host: "my-persistent-domain"
User-Agent: curl/8.4.0
Accept-Encoding: gzip
Content-Type: application/json
Content-Length: 0
X-Forwarded-For: 110.144.26.232,163.116.215.33
X-Forwarded-Host: marten-adapted-dane.ngrok-free.app
X-Forwarded-Proto: https

Note

You'll note that we have some additional "X-Forwarded-..." headers added to out request by Ngrok.

Set up a Marketplacer webhook

Tip

For our full documentation (including video tutorials) on webhooks, please go to the Marketplacer Developer Portal.

To set up a basic webhook these steps:

1. Log into either the Operator or Seller Portal and select webhooks, then New Webhook
2. Ensure that Enabled is checked
3. Select the entity Type you want to work with, in our case we'll select Advert (aka Product)
4. Enter the URL (this should be the URL for your running Ngrok instance)

Leave everything else with the default values for now, we can come back and edit for our other test scenarios later.

Generating a webhook from Marketplacer

We now want to generate a webhook from Marketplacer, to do so we'll update a Product (Advert), this should generate an Advert Update webhook.

By whatever means you prefer, (API, Portal or CSV Upload) update one of your products. Having done so, navigate back to where you created your webhook and click on the "+" to reveal the webhook send attempts:

Click again on the "+" to look at the response given to Markeptlacer by our app:

Looking at where you webhook endpoint is running, you should also see the request headers sent by Marketplacer:

Accept: */*
Host: my-persistent-url
User-Agent: Marketplacer Client/1.0.0
Accept-Encoding: gzip
Content-Type: application/json
traceparent: 00-6625ef040000000029503a352a27b034-1890022d9505b303-01
tracestate: dd=s:1;t.dm:-1
Content-Length: 176
Idempotency-Key: YmVzdGZyaWVuZGJhemFhcjoyNTUy
Marketplacer-Sequence: 2552
Marketplacer-Vertical: bestfriendbazaar
X-Forwarded-For: 13.54.180.168
X-Forwarded-Host: my-persistent-url
X-Forwarded-Proto: https

This concludes the primary set up. The following sections provide some tips on how to test the remaining endpoints and the scenarios they represent.

/webhook/auth

In this example we expect to pass a header (auth-key) along with a key value to identify the webhook to the endpoint - the code performs a simple equivalence check between the key value passed in the header, and the value we have stored as config (as a .NET user secret).

So to get this working we need to add a user secret to our project, to do so, at a command prompt:

• Navigate to the project directory (webhook-endpoint-dotnet)
• Type:

dotnet user-secrets set "auth-key" "0aacc3d70e2749509c47ec6fa55a2282"

This will store an auth-key with a value of 0aacc3d70e2749509c47ec6fa55a2282 in our local secrets repository. This value will then get read in to the .NET configuration layer that we can access from our code.

Tip

While this is just a test project, and we could conceivably get away with just hard-coding the key value, (and not leverage user-secrets) it's good practice to get into this way of working. To learn more about user secrets - refer to this article.

To exercise this endpoint directly using curl you would call it as follows:

curl -XPOST -H 'auth-key: 0aacc3d70e2749509c47ec6fa55a2282' 'https://localhost:7067/webhook/auth'

Assuming you supply the correct header and value, you should get a HTTP 200 OK response, otherwise you will get a HTTP 401 Unauthorized response.

Setting up headers in Marketplacer

To add an equivalent header in Marketplacer, navigate to where you created your webhook, and select the pencil icon (✏️) to edit.

In the Headers section you can add the header as a JSON object, so in this example we would add:

{
  "auth-key": "0aacc3d70e2749509c47ec6fa55a2282"
}

Tip

Don't forget to update the URL to target the /webhook/auth endpoint too.

Updating an Advert should continue to exercise the webhook.

Note

Please refer to our Webhook Retry Policy to understand how we deal with non HTTP 200 OK responses back from a Webhook endpoint such as out test app.

/webhook/errors

This endpoint will respond back to the requester with a HTTP response code of your choice. This can be useful to help you understand how the Marketplacer webhook system handles non HTTP 2xx responses, specifically from a retry perspective.

Note

You can read more about this in the Webhook Retry Policy.

To exercise this endpoint you simply add another header with the following details:

• Header: http-response-code
• Header value: 404 (as an example)

A direct curl request would look as follows:

curl -XPOST -H 'http-response-code: 404' 'https://localhost:7067/webhook/auth'

And you would configure the header in the Markerplacer webhook with the following JSON object:

{
  "http-response-code": 404
}

Tip

Don't forget to update the webhook endpoint to target: /webhook/errors

/webhook/hmac

Note

HMAC is currently only available on Operator webhooks, Sellers don't currently have access to this feature. For a more detailed description on HMAC and webhooks, refer to this article.

To demonstrate HMAC using our app, we need to do the following:

1. Add a HMAC key to your webhook definition (this can be anything you like, but the more complex the better).
2. Update the webhook endpoint to target: /webhook/hmac
3. Store the HMAC key (used in step 1) as a user secret so that our app can retrieve it from configuration securely. To do so, simply enter the following at a command prompt (ensure you are "in" the webhook-endpoint-dotnet project directory):

dotnet user-secrets set "hmac-secret" "<your-hmac-secret>"

Replace <your-hmac-secret> with your selected value.

When you next update an advert and generate webhook, a header (Marketplacer-Hmac-256) will be sent with the request, the value of this header is a hash that has been generated with:

• The SHA-256 hash function
• The full body payload of the webhook
• The HMAC secret

As all these items are also available to our app, we can attempt to recreate the hash and compare it with the header value we received - thus proving (or disproving) that the webhook payload can be trusted.

/webhook/sequencing

Important

This endpoint requires a Sqlite database, so please ensure you have run step 3 (database migrations) of the .NET set up instructions above.

When this endpoint is targeted by a Marketplacer webhook, it reads the value of the Marketplacer-Sequence header and stores it (and the other attributes listed below) in database table called WebhookEvents:

• id: This is the auto-generated table primary key, in this case it is a sequential integer incremented by 1
• Sequence: This is the value of the Marketplacer-Sequence header. It represents the sequence in which the webhook event was generated at source, and can be used to order, or re-sequence, webhook events that may have arrived at the target out of their original sequence.
• WebhookId: This is the unique, Marketplacer assigned id for the webhook event
• WebhookPayload: This is the full serialized JSON payload of the webhook

We make use of this sequencing in the /webhook/payload-diff endpoint.

/webhook/payload-parse

Important

This endpoint requires a Sqlite database, so please ensure you have run step 3 (database migrations) of the .NET set up instructions above.

This endpoint extends the /webhook/sequencing endpoint by extracting specific values from the webhook payload and storing them in the WebhookEvents table, those values are:

• WebhookEventType: Wether this is a create, update of destroy event
• WebhookObjectType: Relates to the object that the webhook is generated from, e.g. Advert, Seller, Invoice etc.
• WebhookObjectId: This is the unique, Marketplacer assigned id of the object

/webhook/payload-diff

Important

This endpoint requires a Sqlite database, so please ensure you have run step 3 (database migrations) of the .NET set up instructions above.

This endpoint compares the payload received for a given WebhookObjectId with the last payload received for that same object, (if any), and determines the differences between the payloads. This helps webhook consumers (such as this app) to determine what has changed. We use the Sequence value to ensure we get the last event for the object.

Note

On a related, yet different subject, you can read about webhook deduplication here, which if enabled, can eliminate redundant webhooks events, (i.e. webhook events that have identical payloads).

Future enhancements

• Hand off webhook processing to a background worker