seojuice
Web Crawler

SEOJuice-SearchBot

Last updated: June 2026

SEOJuice-SearchBot is the web crawler operated by SEOJuice. It visits public web pages to build a graph of how the web links together — which pages reference which — so we can power marketing-intelligence insights for our customers. It is a well-behaved, fully identified crawler. It is not used to collect personal data, and it is never used for any malicious activity.

We believe automated crawling should be transparent. SEOJuice-SearchBot always identifies itself, follows the Robots Exclusion Protocol (robots.txt), honours every blocking directive a site sets, and states — directly in its User-Agent — how to control it. This page is that explanation.

How to identify our crawler

Every request from SEOJuice-SearchBot carries this User-Agent. The + URL links back to this page so any webmaster can find out exactly what we are and how to control us. 

SEOJuice-SearchBot/1.0 (+https://seojuice.io/bot)

The token to match in robots.txt is SEOJuice-SearchBot. We never disguise our crawler as a web browser and never rotate or spoof our identity. (seojuice.io/bot forwards to this page.)

How we crawl responsibly

  • We obey robots.txt. We follow the standard directives — User-agent, Disallow, Allow and Crawl-delay — interpreted the same way Googlebot interprets them.
  • You don't need a rule just for us. We honour your wildcard User-agent: * rules, and we also respect the directives you set for the major search crawlers — if a page is disallowed for Googlebot, we treat it as disallowed for us and won't crawl it.
  • We only fetch public pages. We never attempt to reach pages behind logins, paywalls or any other access control.
  • We always identify ourselves. Every request carries the User-Agent above — no browser impersonation, no rotating disguises.
  • We crawl gently. We rate-limit our requests and respect Crawl-delay so we don't put load on your server.
  • We honour your choices. If you block us we stop, and we re-check your robots.txt regularly so your changes take effect quickly.

How to block SEOJuice-SearchBot

You may already be blocking us without any extra work: SEOJuice-SearchBot honours User-agent: * rules and also respects the directives you set for the major search crawlers, so if a page is disallowed for Googlebot we won't crawl it either.

To control SEOJuice-SearchBot explicitly, add one of the following to the robots.txt file at the root of your domain (for example https://example.com/robots.txt).

Block our crawler from your entire site:

User-agent: SEOJuice-SearchBot
Disallow: /

Block only specific sections:

User-agent: SEOJuice-SearchBot
Disallow: /private/
Disallow: /checkout/

Once we next fetch your robots.txt, we stop crawling the disallowed paths. There is nothing else to do — no account and no form. If you need us to stop immediately, email us below.

Questions or concerns

If you have questions about SEOJuice-SearchBot, want to report a problem, or need us to stop crawling right away, email hello@seojuice.com and we'll respond promptly.