GotAltBot: what GotAlt's web fetcher reads, and how to block it

GotAltBot is the web fetcher behind GotAlt's accessibility checks. It reads a page when a person asks GotAlt to check it, and reads pages on its own for the free scan's extra pages, sitemap discovery, weekly monitoring and GotAlt's own research. A robots.txt rule addressed to GotAltBot by name is always respected.

Published . Last reviewed .

Its user agent

Every request GotAltBot makes carries this user agent:

GotAltBot/1.0 (+https://gotalt.com/bot; accessibility checker)

In robots.txt, address it as GotAltBot. The name is matched without regard to case.

What it fetches, and when

GotAltBot makes two kinds of request. Some come from a person asking GotAlt to check one page, and GotAltBot treats those much as a browser treats a person opening that page. The rest are pages GotAlt reads on its own. Before it reads any page on a site, GotAltBot checks that site's /robots.txt.

What GotAltBot requests, what starts it, and which robots.txt rules apply
What starts it What GotAltBot requests robots.txt rules that apply
A visitor runs the free scan The page they entered Groups addressed to GotAltBot by name
The same free scan Up to two more pages from the same site, picked from the first page's links All of them: the GotAltBot group, or the User-agent: * group if there is none
A visitor runs the deep AI audit The free scan above, then up to six of the page's images that have alt text Groups addressed to GotAltBot by name, for the page and for each image's own address
A visitor runs the alt text checker The page they entered and up to six of its images Groups addressed to GotAltBot by name; the images follow their page
A visitor enters an address in the heading checker or the link text checker The page they entered Groups addressed to GotAltBot by name
A customer adds a site to monitoring, or asks for its pages from the sitemap The home page (to see whether the site answers at its www or bare address), and the sitemap files on the site itself All of them
Monitoring of the pages a customer has added Each page once a week, plus a first check when it is added, an extra check the customer can ask for once a day per site, and retries after a failed check, with up to six of its images each time All of them; the images follow their page
GotAlt's own research One page of a site GotAlt chose, to research the market or to show a prospective customer a finding on their own site, and up to six of its images that have an alt attribute All of them; the images follow their page

How it applies robots.txt

A page a person asked us to check

When a person submits a page in the free scan, the deep AI audit, the alt text checker, or the address box of the heading and link text checkers, GotAltBot fetches that page and the images on it the way a browser would, because a person asked for that page. Wildcard (User-agent: *) rules do not apply to that single requested fetch. Rules in a group addressed to GotAltBot by name are always respected, and they are checked again at every redirect. If the site's robots.txt cannot be read, the requested page is still fetched.

Pages GotAlt finds or re-checks on its own

The free scan's extra pages, sitemap discovery, scheduled monitoring and GotAlt's own research follow robots.txt in full: the group addressed to GotAltBot if the file has one, otherwise the User-agent: * group. Every redirect is checked again against the robots.txt of the address it leads to.

If a site's robots.txt returns a server error (a 5xx status), times out, cannot be reached, or redirects more than four times, GotAltBot does not crawl that site on its own until it can read the file. A 404 or any other 4xx status means there are no restrictions.

Images

Images are fetched as part of their page and follow the page's decision. For the alt text checker, monitoring and GotAlt's own research, GotAltBot does not check robots.txt separately for an image's own address, including an image served from another host such as a content delivery network.

The deep AI audit is the exception. It receives its list of images from the visitor's browser rather than from a page GotAltBot read, so it also checks each image's own address, and every redirect, against rules addressed to GotAltBot by name, and skips an image those rules cover. Wildcard rules do not stop these images, as they do not stop the page.

How rules are read

  • The longest matching rule wins, and an Allow rule wins a tie.
  • * matches any run of characters, and a final $ marks the end of the address. Rules are matched against the path and the query string, so Disallow: /*?sort= covers /products?sort=price.
  • GotAltBot reads the first 512 KiB of a robots.txt file, and ignores a line that limit cuts in two.
  • It keeps a site's robots.txt for up to an hour, so a change reaches it within an hour. A file it could not read, or that answered 429 Too Many Requests, is tried again after five minutes.
  • It does not read Crawl-delay. Its own pacing is described below.

How to block GotAltBot

To stop GotAltBot reading any page of your site, including pages a person asks GotAlt to check, add this group to your robots.txt:

User-agent: GotAltBot
Disallow: /

This group covers requests for addresses on your site. An image of yours shown on another site's page is fetched as part of that page and follows that page's robots.txt, not yours, except in the deep AI audit, which also checks the image's own address as described above.

To keep it out of one part of your site, disallow that path in the same group:

User-agent: GotAltBot
Disallow: /checkout/

A User-agent: * group on its own stops the pages GotAlt reads on its own, but not a page a person asks us to check. If your wildcard group disallows everything and you want GotAlt to monitor your own site, let GotAltBot in by name:

User-agent: GotAltBot
Allow: /

If you pay for monitoring and your robots.txt blocks GotAltBot from a monitored page, that page isn't checked and your account page says so. It still counts toward your page count until you remove it or allow GotAltBot.

What it does not do

  • It does not run JavaScript. It reads the HTML your server sends, so content a script adds after the page loads is not seen.
  • It does not fill in or submit forms, sign in, or keep cookies. Its requests are ordinary GET requests.
  • It does not fetch a page's stylesheets, scripts or fonts. It reads robots.txt, the page itself, sitemaps, and the images the checks above describe.
  • It follows at most four redirects for one request, and only to public addresses: it refuses private and internal network addresses.

How fast it goes

  • The free scan reads at most three pages of a site, one at a time.
  • Monitoring spaces one site's pages about six seconds apart in each scheduled run.
  • If a site answers 429 Too Many Requests, or 503 with a Retry-After header, monitoring pauses that site: for five minutes at first, doubling each time it happens again up to two hours, or for as long as Retry-After asks if that is longer, up to six hours.

Contact

If GotAltBot is causing a problem on your site, email hello@gotalt.com with your site's address and the time of the requests. We reply within two working days (Monday to Friday). Our terms set out the same robots.txt policy.

Questions about GotAltBot

Why did GotAltBot read a page my robots.txt disallows for all bots?

Because a person asked GotAlt to check that page. Wildcard rules do not apply to that single requested fetch, in the same way they do not stop a person opening the page in a browser. To stop it, add a group addressed to GotAltBot by name with Disallow: /.

Does GotAltBot run JavaScript or submit forms?

No. It reads the HTML your server sends and makes ordinary GET requests. It does not run scripts, fill in or submit forms, sign in, or keep cookies.

How soon does a change to my robots.txt take effect?

Within an hour. GotAltBot keeps a site's robots.txt for up to an hour before reading it again.

What happens while my robots.txt is down?

If it returns a server error, times out or cannot be reached, GotAltBot does not crawl your site on its own, and tries the file again after five minutes. A page a person asks GotAlt to check is still fetched. A 404 means there are no restrictions.

Does GotAltBot follow Crawl-delay?

No. It does not read Crawl-delay. Monitoring spaces one site's pages about six seconds apart and pauses a site that answers 429, or 503 with a Retry-After header.

See what GotAlt checks on your site

Run a free scan to see which problems automated checks find, or look at a real report first.

Scan my site for free See a real report first