The following example shows you how to allow blank referers for social network crawlers when HTTP Referer protection is turned on. Social networks use different user agents to access content, so we need to allow blank referer for these user agents only. Any other user agents will receive a 403 Forbidden status code, as well as requests with non-listed referers.

EdgeRules Criteria and Features

NGINX Configuration Block

location / {
	valid_referers *.example.com;
	set $flag “1”;
	if ($http_user_agent ~* (Twitterbot|Pinterest|facebookexternalhit|LinkedInBot|Google) {
		set $flag “0”;
	}
	if ($http_referer ~* ^$) {
		set $invalid_referer $flag;
	}
	if ($invalid_referer = 1) {
		return 403;
	}

cURL Examples

In the examples below, I’ll show you the desired behavior via the cURL command, passing the appropriate request parameters so we can mimic different use cases.

No referrer and no user agent should return 403:

$ curl -I http://cdn.example.com/MaxCDN.png
HTTP/1.1 403 Forbidden
Date: Thu, 10 Mar 2016 19:10:23 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Server: NetDNA-cache/2.2

user agent “Google” should be allowed:

$ curl -I http://cdn.example.com/MaxCDN.png -A “Google”
HTTP/1.1 200 OK
Date: Thu, 10 Mar 2016 19:10:47 GMT
Content-Type: image/png
Content-Length: 46538
Connection: keep-alive
Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT
ETag: “56e1bbca-b5ca”
Server: NetDNA-cache/2.2
X-Cache: HIT
Accept-Ranges: bytes

user agent “Twitterbot” should be allowed:

$ curl -I http://cdn.example.com/MaxCDN.png -A “Twitterbot”
HTTP/1.1 200 OK
Date: Thu, 10 Mar 2016 19:10:53 GMT
Content-Type: image/png
Content-Length: 46538
Connection: keep-alive
Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT
ETag: “56e1bbca-b5ca”
Server: NetDNA-cache/2.2
X-Cache: HIT
Accept-Ranges: bytes

user agent “Pinterest” should be allowed:

$ curl -I http://cdn.example.com/MaxCDN.png -A “Pinterest”
HTTP/1.1 200 OK
Date: Thu, 10 Mar 2016 19:10:58 GMT
Content-Type: image/png
Content-Length: 46538
Connection: keep-alive
Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT
ETag: “56e1bbca-b5ca”
Server: NetDNA-cache/2.2
X-Cache: HIT
Accept-Ranges: bytes

user agent “facebookexternalhit” should be allowed:

$ curl -I http://cdn.example.com/MaxCDN.png -A “facebookexternalhit”
HTTP/1.1 200 OK
Date: Thu, 10 Mar 2016 19:11:05 GMT
Content-Type: image/png
Content-Length: 46538
Connection: keep-alive
Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT
ETag: “56e1bbca-b5ca”
Server: NetDNA-cache/2.2
X-Cache: HIT
Accept-Ranges: bytes

user agent “LinkedInBot” should be allowed:

$ curl -I http://cdn.example.com/MaxCDN.png -A “LinkedInBot”
HTTP/1.1 200 OK
Date: Thu, 10 Mar 2016 20:09:31 GMT
Content-Type: image/png
Content-Length: 46538
Connection: keep-alive
Last-Modified: Thu, 10 Mar 2016 18:24:10 GMT
ETag: “56e1bbca-b5ca”
Server: NetDNA-cache/2.2
X-Cache: HIT
Accept-Ranges: bytes