Custom Integration for DSR

The custom integration allows you to extend the DSR handling workflow with your own code. This is especially useful for applying custom business logic, handling internal home-brew systems or preventing external connections to internal databases. This is extremely easy to implement with your own API endpoint (such as a cloud function) or even no-code automation tools, such as Zapier or Integromat.

The custom integration works by allowing you to register API endpoints that will get called as part of the DSR workflow. Just like Mine's built-in integrations, custom integration supports any request type (Delete, Copy...), allows previewing user data, and supports advanced operations such as handling failures and performing long-running async operations.

The custom integration supports the following operations. Each operation requires an endpoint URL in order to be used:

Note: while Preview is optional, at least one of the copy or delete endpoints has to be defined.

Adding Custom Integrations

Custom integration is available for custom systems in your data inventory. To more than one custom integration simply add multiple custom systems.

When multiple custom integrations are used, the integrationId field is used for updating the status of a specific integration.

In the following sections we will describe each operation and how it works.

Delete

When a delete workflow is executed, MineOS makes an API call to the URL set in the Delete URL field. The HTTP request is of the following form:

POST https://example.com/endpoint HTTP/1.1
X-Mine-Signature: 4r4ClKnSeGWvTGQzMCUcEEK9uF528mvOlN/jtKMQ==
Content-type: application/json

{
  "EventId": "123456789abcdefghijklm",
  "EventType": "WebhookSlim",
  "isTest": "false",
  "ticketInfo": ...
}

The receiving endpoint is expected to return one of the following responses:

1. 200 OK - Indicates the deletion process has been completed successfully and will be marked as 'completed' (synchronous flow).
2. 202 Accepted - Indicates the deletion request has started processing and will be marked as 'in progress' (async flow). Once finished, you should use the Update custom integration status [v2] endpoint to Completed/Failed/etc.

Failing to respond with 200/202, or returning any other non-2xx status, will cause MineOS to keep retrying the request.

Handling success and failure

When using the Update custom integration status [v2] endpoint it's important to use the status field properly. In general:

• Completed: The result was completed successfully, and MineOS should consider this DSR as completed, meaning no further operation/retry is necessary.
• UserNotFound: The requested user (or his data) was not found. Since its a perfectly valid state this is treated the same as 'Completed'.
• CannotDeleteData: The requested user and his data was found, but the system is not allowed to delete it. This is treated the same as 'Failed'.
• Failed: means the result of the operation is not as intended, and the system should not consider this DSR as completed. Failed usually means an error that cannot be retried and requires user intervention.

Here are a few examples that demonstrate proper use of the status field:

Example 1: A request of type delete, but no user data/id was found.

In this example, you should use status=UserNotFound. The reason is that the DSR can (and should) be considered handled. If a user requested his data to be deleted and no data exists - that is perfectly fine.

Example 2: A request of type delete, but some user data has to be kept

In this example, user data was deleted, with the exception of some data that had to be retained due to other business/compliance requirements. As this is a perfectly acceptable way to handle deletion requests and the DSR can be considered as done, you should use status=Completed.

Example 3: A request of type delete, but the user has future reservations or an active subscription and cannot be deleted

In this example, user data should not be deleted as a business decision. The DSR cannot be honored and should not be considered as done, you should use status=CannotDeleteData.

Example 4: A request of type delete, but some user data failed to delete

In this example, user data was attempted to be deleted but some of it failed. This can happen because of a network error accessing a required endpoint, database authentication error or a number of other reasons that prevented required data from being deleted. In this example the DSR should not be considered as completed, and the user should use status=Failed.

Copy

When a copy workflow is executed, MineOS makes an API call to the URL set in the Copy URL field. The HTTP request is of the following form:

POST https://example.com/endpoint HTTP/1.1
X-Mine-Signature: 4r4ClKnSeGWvTGQzMCUcEEK9uF528mvOlN/jtKMQ==
Content-type: application/json

{
  "traceId": "123456789abcdefghijklm",
  "integrationId": "0c36xnykgewwbzfukh1jkq",
  "isTest": true
  ...
}

The receiving endpoint is expected to return one of the following responses:

1. 200 OK - Indicates the data collection process has been completed successfully and will be marked as 'completed' (synchronous flow). The response contains any JSON data that was collected.
2. 202 Accepted - Indicates the data collection request has started and will be marked as 'in progress' (async flow). Once finished, you should use the Update custom integration status endpoint to update the integration status to Completed/Failed/etc and provide the data that was collected.

Failing to respond with 200/202, or returning any other non 2xx status, will cause MineOS to keep retrying the request.

Providing Collected Data - Sync flow

The receiving endpoint is expected to return 200 OK with a JSON response body including any user data that should be collected as part of the copy workflow. The response JSON can be of any format, it has no schema. Any information returned by this endpoint is collected by the system and sent to the requesting user.

Providing Collected Data - Async Flow

Collected data is sent to MineOS when calling the Update custom integration status endpoint, in one of two ways:

1. JSON data
2. File URL

Preview and User Search

When a any workflow is about to get executed, MineOS makes an API call to the URL set in the Preview URL field. The HTTP request is of the following form:

POST https://example.com/endpoint HTTP/1.1
X-Mine-Signature: 4r4ClKnSeGWvTGQzMCUcEEK9uF528mvOlN/jtKMQ==
Content-type: application/json

{
    "traceId": "123456789abcdefghijklm",
    "integrationId": "0c36xnykgewwbzfukh1jkq",
    "isTest": true
    ...
}

The receiving endpoint is expected to return 200 OK with a JSON response body that includes a summary of user data records found. This data will not be shared with the requesting user, and is used as internal information to help you choose how to handle the request.

The response JSON must satisfy match the following structure:

{
  "records": [
    {
      "name": "firstRecord",
      "properties": [
        {
          "name": "fieldName",
          "value": "fieldValue"
        },
        {
          "name": "anotherName",
          "value": "anotherValue"
        }
      ]
    },
    ...
  ]
}

📘

Notes

• You can returns as many records as you want
• You can only return up to 3 properties for each record.
• All property types should be string.
• The request object does not always exist in a preview request.

Request Payload

Every time MineOS makes an HTTP call to your endpoint, the following JSON payload is sent as the request body:

{
    // usefull for logging
    "traceId": "123456789abcdefghijklm",
    // used for the CustomIntegrationStatus endpoint
    "integrationId": "0c36xnykgewwbzfukh1jkq",
    // indicates this is just a test event, actual processing should be skipped
    "isTest": true,
    // the request object is not always available:
    // its available when handling a DSR but not handling when handling a user search
    "request": {
        // used for the CustomIntegrationStatus endpoint
        "id": "ABCDEFGHIJKLMNOPQRSTUV",
        // type of request. Possible values: Delete,GetCopy,DoNotSell,Undetermined,RightToEdit,DoNotMail
        "type": "Delete",
        // the channel from which the request was sent. Possible values: Form,MineApp,Api,EmailForwarding,Manual
        "source": "Form",
        // the main domain of the workspace to which the request was sent
        "domain": "test.com",
        "createdAt": "2023-07-28T11:35:21.7528109Z"
    },
    "userInfo": {
        "name": "Test User",
        "email": "[email protected]",
        "isVerified": false,
        "countryOfResidence": "TestCountry",
        // fields set through UpdateMetadata endpoint
        "customFields": {
            "testCustomKey": "Custom value"
        }
    }
}

Logging And Storing Information

At any time while processing a request, you might want to store information on the request:

1. Logging/documenting important business information, for example: the type of customer, the underlying system etc. This can be done with Add a note API. In the UI this information is visible in the notes section on the right side.
2. Storing metadata/state information, for example: internal user Id, URI of related resources or other metadata about the handling flow. This can be done with Update metadata API. This information is not visible in the UI but can be retrieved with Get metadata API.

Security

When receiving requests to your endpoint, its best to perform a security check before processing the event. There are a number of supported security mechanisms, you can use any one of them, or a combination of more than one:

1. API Key - the requests can include a header with a name and value of your choice. By setting a secret value as a custom header, you can verify in your code that the header contains the value your would expect, similar to using an API Key. This is the best and easiest option.
2. IP Whitelisting - we use static IPs that you can whitelist in your firewall / ACL. See the list of IPs below.
3. Verifying signature - this is another option instead of the API Key. Each request MineOS makes to your endpoint contains a HTTP header called: x-mine-signature. This header contains the event signature, which can be verified to make sure the request originated from your MineOS account. See example code for verifying the signature:

var key = "YOUR VERIFICATION KEY";
var data = "JSON PAYLOAD OF THE EVENT";

var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(key));
var rawSignature = hmac.ComputeHash(Encoding.UTF8.GetBytes(data));
var signature = BitConverter.ToString(rawSignature).Replace("-", "").ToLower();

var key = "YOUR VERIFICATION KEY";
var data = "JSON PAYLOAD OF THE EVENT";

var crypto = require('crypto');
var signature = crypto.createHmac('SHA256', key).update(data).digest('hex');

To test weather your implementation is correct, you can use an online HMAC calculator, like this one.

MineOS IPs:

34.77.7.236
35.187.97.141
34.76.247.90
34.140.71.114