X-Robots-Tag: Control Search Engine Indexing

What is the X-Robots-Tag?

The X-Robots-Tag is an instruction that controls how search engines handle a specific file or page, such as whether it may be indexed or whether search engines should follow the links on it. Unlike the well-known Robots Meta Tag, which is placed in the HTML code of a page, the X-Robots-Tag is transmitted in the so-called HTTP header, i.e., in the technical information that a server sends when a file is called up, before the actual content is loaded.

This difference in the transmission method is the key advantage: While the Robots Meta Tag only works in HTML documents, the X-Robots-Tag can be applied to practically any file type, including PDFs, images, or other documents that do not contain any HTML code.

The Difference from the Robots Meta Tag

Both tools serve the same purpose, namely giving search engines indexing instructions, but they differ in their method:

Robots Meta Tag: Located in the <head> section of an HTML page, e.g., <meta name="robots" content="noindex">. Only works with HTML documents.
X-Robots-Tag: Sent in the HTTP header and works for any file type, including non-HTML files such as PDFs or images.

In simple terms: The Robots Meta Tag is the solution for HTML pages, while the X-Robots-Tag is the more flexible, server-side option for all other cases.

Which Instructions Are Possible?

The X-Robots-Tag supports the same instructions (directives) as the Robots Meta Tag. The most important ones are:

noindex: The page or file should not be included in the search index.
nofollow: Links on the page should not be followed.
noarchive: No cached copy should be offered in the search results.
none: A combination of noindex and nofollow.

An example of how such an instruction looks in the HTTP header:

X-Robots-Tag: noindex, nofollow

When Is the X-Robots-Tag Particularly Useful?

The biggest advantage lies in controlling non-HTML files. Typical use cases include:

Keeping PDF files out of the index: Such as price lists, internal documents, or whitepapers that should be available but not appear in search results.
Excluding images from indexing: If certain graphics should not appear in Google Image Search.
Regulating entire file types: Via server configuration, an instruction can be applied to all files of a specific type, such as all PDFs on a website at once.

The Important Connection with robots.txt

A crucial point that is often done incorrectly: For Google to be able to read the X-Robots-Tag (e.g., a noindex), the file must be crawlable. If the same file is simultaneously blocked from crawling via robots.txt, Google cannot see the instruction in the HTTP header because the file is not retrieved at all.

This means specifically: If you want to reliably keep a file out of the index, you must not block it via robots.txt but must leave it crawlable and provide it with a noindex via the X-Robots-Tag. This connection is identical to that of the Robots Meta Tag and is one of the most common technical SEO mix-ups.

What to Watch Out For

Server access required: The setup is usually done via server configuration (e.g., the .htaccess file for Apache or the configuration for Nginx). This requires technical knowledge or support from the hosting provider.
Test carefully: Since the instruction takes effect on the server side and often affects entire file types, an error can quickly unintentionally remove many files from the index.
Check via Search Console: The URL inspection tool in Google Search Console can be used to verify whether the instruction is correctly recognized.

Conclusion

The X-Robots-Tag is a powerful tool in technical search engine optimization that allows you to control the indexing of content via the HTTP header. Its major advantage over the Robots Meta Tag is its flexibility: It works for any file type, including PDFs and images. The key to proper use is understanding that the affected file must remain crawlable for the instruction to be read. Due to the required server access and its far-reaching effects, the X-Robots-Tag should be used with care and thoroughly checked after implementation.