Configuring proxy endpoints

Adding DataUnlocker to your website consists of 3 steps: configuring proxy endpoints, adding a script and running a health check to ensure that you have set things up properly.

However, this simple setup will require some periodical maintainance on your side, unless you're a very small website and ad blocker community doesn't even care about you. Check the automation concepts to learn about the fully-managed setup, which you can perform after going through the basic setup below.

The concept of proxy endpoints

Proxy endpoints are basically the URLs on your domain which proxy traffic to DataUnlocker servers. Every blocked request will be retried and routed via the proxy endpoint used in DataUnlocker's script:

• https://random-hash.yourdomain.com/... - DNS proxy endpoints
• https://yourdomain.com/random-hash/... - path-based proxy endpoints

Proxy endpoints are designed to be dynamic and easy to replace, as this is pretty much the only "single point of failure" of the DataUnlocker set up, a thing which ad blocker community can block.

At the first step of the installation guide, you will be asked to set up a proxy endpoint, which can be either:

• DNS-based (default, easier to set up but also easier to become blocked)
• Path-based (harder to set up but almost impossible to be blocked)

Read more about the difference between the two below.

You can also add multiple proxy endpoints, but normally you would only need one, as only one proxy endpoint is used by DataUnlocker's script at a time. You (or DataUnlocker, in case of the automated setup) will only need to create a new proxy endpoint in case the old one is blocked.

Differences between DNS and path-based proxy endpoints

TL;DR We recommend using path-based proxy endpoints unless you're serving critical website resources on its subdomains disallowing ad blocker community to block all your DNS proxy endpoints with *.yourwebsite.com mask. DNS proxy endpoints are still good for evaluation or small websites which are unlikely to attract much of the community's attention.

The main difference between the two proxy endpoints types comes from the way they're set up:

DNS-based proxy endpoints are subdomains of your website, pointing to DataUnlocker's servers. For example, rand0m7hash.yourwebsite.com.

• DNS proxy endpoints are easier to set up, as at minimum you need to just add/remove a DNS record on your domain.
• They are also much easier to become blacklisted, as for instance, if your website doesn't have any critical resources served on a subdomain, at some point ad blocker community can blacklist all your proxy endpoints permanently with *.yourwebsite.com mask. Ensure you have some resources (scripts) served on a subdomain to prevent this, or configure path-based proxy endpoint instead.
• ⚠️ If you set up DNS proxy endpoint without your own proxy in front of DataUnlocker's servers, for instance, AdGuard's DNS crawlers or the community may find your domain quickly and block it. We recommend you to always place a reverse proxy in front of DataUnlocker's servers.

Path-based proxy endpoints are subpaths of your website pointing to DataUnlocker's servers. For example, yourwebsite.com/rand0m7hash.

• For path-based proxy endpoints you have to use your own reverse proxy, leaving no chance for the ad blocker community to find you via the DNS network.
• It is impossible to block all path-based proxy endpoints URLs since they all collide with your website resources.
• Your own reverse proxy makes it resistant to any DNS blockers like Pi-hole, as blocking your entire website is not feasible.
• When using automation, unlike DNS proxy endpoints, path-based proxy endpoints can be replaced in seconds.
• Path-based proxy endpoints can be harder to set up comparing to DNS-based proxy endpoints, as you need to configure your servers to proxy the traffic to DataUnlocker servers via the URL prefix.

Proxy endpoint's URL (regardless of the type) is the only static recognizable part of the URL, while the rest of the URL being unpredictably encoded: rand0m7hash.yourwebsite.com/xOs7m2PlwnS8qA91.... Use automation to periodically update this URL avoiding blacklists.

Setting up a DNS-based proxy endpoint

The setup guide will suggest a new DNS record you will need to add to your domain at the very beginning. This is usually done on the website where you have purchased your domain. The exact steps for adding a DNS record vary between domain registrars, and we suggest you to look at some common cases to understand how DNS is configured.

Add DNS records exactly as the setup guide suggests, and you should get a working endpoint soon:

Note the following:

• ⏳ It takes around 5 minutes for the new DNS record to propagate through the DNS network and reach Google DNS servers (which is used by DataUnlocker). For some providers it may take a bit longer. You can use different tools to check whether your new DNS record started to propagate.
• ⚡ If you're using CloudFlare for managing DNS records, please ensure the proxy is TURNED ON for this DNS record (an orange-cloud icon is not orange).
• ✅ We highly recommend setting up your own reverse proxy (like CloudFlare DNS proxy) in front of DataUnlocker's proxy servers. Since DataUnlocker's IP addresses are tracked, your domain might get to blacklists quickly.

Setting up a path-based proxy endpoint

If you decide to use the path-based proxy endpoint, you will need to run your own reverse proxy back end like this one, or utilize existing tools like Cloudflare workers etc. You also need to ensure that your reverse proxy works properly:

• It passes X-Forwarded-For header to all proxied requests.
• It doesn't modify any existing headers nor path (the path prefix must stay).

ℹ️ When setting up path proxy endpoint, note that the target hostname *.ddns.dataunlocker.com doesn't change between proxy endpoints, so you can safely hardcode it in your proxy code. The pathname prefix (urlKey in yourwebsite.com/urlKey/*), however, can change, if urlKey is added to blacklists.

Setup example for Nginx

Here's one of the examples of setting up a reverse proxy if you're using nginx, which should make yourwebsite.com/urlKey/* get proxied to https://duKey.ddns.dataunlocker.com/*. Note that in this example urlKey is a part of URL DataUnlocker asks to reserve on yourwebsite.com, which will redirect HTTP request to a specific duKey.ddns.dataunlocker.com.

location /urlKey/ { # keep the trailing slash here
    proxy_set_header X-Forwarded-For $http_cf_connecting_ip;
    proxy_set_header Host $http_host;
    proxy_buffering off;
    proxy_pass https://duKey.ddns.dataunlocker.com/urlKey/; # keep the trailing slash here
}

Setup example for CloudFlare (CloudFlare Workers)

If you are using CloudFlare and are proxying traffic through CloudFlare servers (an "Orange Cloud" at the DNS record), you can initialize CloudFlare Workers to work with path proxy endpoints quite simply.

First, ensure the "Orange Cloud" is enabled for your domain. Thus, CloudFlare will be able to control paths on your domain.

CloudFlare worker is a piece of code which runs on all edge locations of CloudFlare, allowing to, for instance, proxy traffic further according to the rules you define. To setup DataUnlocker's path proxy endpoint with CloudFlare workers, we need the following:

1. Create a CloudFlare Worker which will proxy any received traffic to DataUnlocker servers.
2. Create the "HTTP Route" as requested by DataUnlocker, and point it to the previously created worker.
3. Test the endpoint on DataUnlocker and ensure it's green!

Create a CloudFlare Worker

We create a worker which just proxies any received traffic down to DataUnlocker's servers, as suggested by the guided setup in DataUnlocker. First, visit the "Workers" section in CloudFlare and add a new worker:

Then, add a service in this worker:

Configure this service:

• Name it, for example, "dataunlocker-proxy".
• Select "HTTP handler" as a starter for this worker.
• Click "Create service".

Then, click the "Quick Edit" button near the newly created worker:

Insert the following code to the worker (JavaScript):

const DU_PROXY_HOSTNAME = 'XXXXXXXXX.ddns.dataunlocker.com';

addEventListener('fetch', (event) => {
  const url = new URL(event.request.url);
  handleRequest(event, url);
});

async function handleRequest(event, url) {
  const [, protocol, , path] = url.toString().match(
    /^([^:]+):\/\/([^\/]+)(\/?.*)/
  );
  const {headers, method, body} = event.request;

  event.passThroughOnException();
  event.respondWith(
    fetch(
      `${protocol}://${DU_PROXY_HOSTNAME}${path}`,
      { headers, method, body }
    )
  );
}

Replace XXXXXXXXX.ddns.dataunlocker.com with the target domain suggested by DataUnlocker during the setup of the proxy endpoint. You can hardcode it, since this hostname doesn't change between proxy endpoints of the single property.

Then, click "Save and Deploy".

Create the HTTP Route

Now, go to your website on CloudFlare and select "Workers" from the left menu. Click "Add Route" in HTTP Routes:

Use the UI to configure the newly added HTTP Route:

1. Set the "Route" to *yourwebsite.com/XXXXXXXXX/*, where XXXXXXXXX is urlKey (path prefix) DataUnlocker asks you to configure, and yourwebsite.com is the hostname of your website.
2. In "Service", choose the worker service you have created previously, dataunlocker-proxy.
3. Pick the environment, which is usually Production.
4. Click "Save".

Congrats, now you have CloudFlare worker set up!

Health check the path proxy endpoint

Now go to DataUnlocker and click "Test" on the proxy endpoint you were just creating the setup for. It should give a healthy result!