Write a robots.txt by the RFC 9309 rules, then test a URL against it and see which rule wins and why.
Initializing in your browser…
Write SEO, Open Graph and X card tags with every value escaped for an attribute, checked against the protocol, and the cut shown per platform.
Build a Subresource Integrity attribute, with more than one hash if you want, and check an existing one against the file it is meant to pin.
Generate passwords from the Web Crypto generator by rejection sampling, with the entropy stated exactly from the alphabet, the cost of every rule computed, and crack times against four named attacker models
A WordPress site is not being crawled properly and someone suggests the rules are in the wrong order. You paste the live file in and test the URL that matters.
The live robots.txt
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: /sitemap.xml
Testing /wp-admin/admin-ajax.php as Googlebot
Allowed
Allow: /wp-admin/admin-ajax.php is the most specific match at
24 octets. Order in the file makes no difference: the standard
says the most specific match wins, not the first one.
EVERY RULE THAT MATCHED, MOST SPECIFIC FIRST
Allow: /wp-admin/admin-ajax.php (24 octets, wins)
Disallow: /wp-admin/ (10 octets)
Testing /wp-admin/options.php
Blocked
Disallow: /wp-admin/ is the most specific match at 10 octets.
[error] Sitemap: /sitemap.xml is not a full URL
A robots.txt is fetched on its own, so there is no page for a
crawler to resolve a path against, and a relative value is
simply dropped.
[warning] Disallow: /wp-admin/ publishes that path rather than
protecting it.The order was never the problem, and 24 against 10 is why. RFC 9309 section 2.2.2 says "The most specific match found MUST be used. The most specific match is the match that has the most octets", so the Allow wins because it is longer, and swapping the two lines gives exactly the same answer. Type them in either order here and watch the verdict stay the same.
That one rule is the source of most robots.txt confusion, and a great deal of tooling implements the folklore instead. Python's own urllib.robotparser returns the FIRST matching rule, so it answers Blocked for this file: the project tests drive it alongside a second implementation of RFC 9309 written from the document, and the stdlib disagrees with the standard on three of four sample files. Every verdict this page gives is compared against that reference, 5,040 combinations of file, crawler and path in the module tests and 440 more through the browser, with no disagreements.
The real fault in this file is the Sitemap line, and it is silent. Its value has to be a full URL, because a robots.txt is fetched on its own and a crawler has no page against which to resolve /sitemap.xml. The line is syntactically perfect and is simply dropped, so the site looks like it has a sitemap and does not.
The warning underneath is the other thing worth knowing. A Disallow line does not hide anything: robots.txt is served from a fixed public URL that anybody can read, so listing /wp-admin/ tells every reader it exists, and the crawlers that ignore the file will fetch it anyway. It is a request to well-behaved crawlers, not a control.
Build a robots.txt and then test a URL against it: the page says allowed or blocked, names the rule that decided it, and lists every rule that matched with the length that ranked it. The matching follows RFC 9309, which standardised the protocol in 2022 and which most tools still do not implement.
The first is the one that surprises everyone. **The most specific rule wins, not the first one.** Section 2.2.2 says "The most specific match found MUST be used. The most specific match is the match that has the most octets." So `Disallow: /` followed by `Allow: /public` allows `/public/page`, and so does the same pair written the other way round, because order in the file changes nothing at all. Python's own `urllib.robotparser`, which a great deal of tooling is built on, returns the first matching rule instead: the project tests drive both and it disagrees with the standard on three of four sample files.
The second follows from it. **An Allow beats a Disallow of the same length.** Same section: "If an 'Allow' rule and a 'Disallow' rule are equivalent, then the 'Allow' rule SHOULD be used." That is what makes the WordPress convention work, where `Disallow: /wp-admin/` sits above `Allow: /wp-admin/admin-ajax.php`.
The third is about shape. **A group is one or more User-agent lines followed by rules**, section 2.2.1, so two User-agent lines in a row share one rule set. That is how you write one policy for ten AI crawlers without repeating it ten times, and a tool with one agent per block cannot express it. The preset here writes one group with ten User-agent lines rather than ten groups.
The fourth is about the patterns. **Only `*` and `$` are special, and only in the path**, section 2.2.3. There is no regular expression: a dot is a dot and a plus is a plus, and `$` outside the last position matches a literal dollar sign. The page will tell you that `/a.b` does not match `/axb`, because you can type it in and see.
Two directives people rely on are not in the standard at all, and are reported rather than written out silently. `Crawl-delay` was never adopted and Google ignores it outright, saying so in its own documentation; the crawl rate setting in Search Console is the only lever there. `Host` was a Yandex extension for naming a preferred mirror and Yandex retired it in 2021. `Noindex` in robots.txt was an undocumented Google behaviour removed in September 2019 and now does nothing at all.
The `Sitemap` line has its own trap and it is silent. Its value has to be a full URL: RFC 9309 section 2.2.3 gives it as a URI, and a robots.txt is fetched on its own, so a crawler has no page against which to resolve `/sitemap.xml` and simply drops the line. The file still looks fine. This tool asks for your site's address and writes the absolute URL out.
And the thing worth saying loudest: **robots.txt is a request, not a control.** It lives at a fixed public URL that anybody can read, it is obeyed only by crawlers that choose to, and a `Disallow: /admin` line advertises the existence of that path to everyone, including the crawlers that ignore the file. Any path here that looks like it is being hidden gets a warning saying so. A page that must not be public needs authentication; a page that must not be indexed needs a robots meta tag or an X-Robots-Tag header, which means it has to stay crawlable for the instruction to be seen, because a page blocked in robots.txt can still appear in results listed by its URL alone.
Every answer the page gives is checked in the project tests against a second implementation of RFC 9309 written in Python from the document: 5,040 combinations of file, crawler and path in the module tests and 440 more driven through the browser, with no disagreements.
Paste the live file, type the URL, and read which rule won. Usually a longer Allow further down that nobody expected to matter.
One group with ten User-agent lines rather than ten groups, which is both shorter and what the standard describes.
The checks name a relative Sitemap, a Disallow: / left over from staging, and directives no crawler reads any more.
Type the same two rules in both orders and watch the verdict stay the same. The most specific match wins, and the standard says so.
No. RFC 9309 section 2.2.2 says the most specific match wins, and specificity is measured in octets of the pattern. Disallow: / above Allow: /public and Allow: /public above Disallow: / give the same answer for /public/page. You can type both here and see.
Because the Allow is 24 octets and the Disallow is 10, so the Allow is the more specific match. That is the same rule as above, and it is why the WordPress convention works.
Not for Google, which ignores it and says so in its documentation. It is not in RFC 9309 at all. Bing and Yandex read it. For Google the crawl rate setting in Search Console is the only lever.
Because robots.txt is fetched on its own, so a crawler has no page context to resolve /sitemap.xml against. RFC 9309 section 2.2.3 gives the value as a URI, and Google documents that it must be fully qualified. A relative one is simply dropped, and the file still looks correct.
Yes, and it is what the standard describes. Section 2.2.1 defines a group as one or more User-agent lines followed by rules, so several User-agent lines in a row share the rules beneath them.
No. robots.txt is served from a fixed public URL that anybody can read, so a Disallow line tells everyone that path exists, and crawlers that ignore the file will fetch it anyway. Use authentication for anything that must not be public.
Not reliably. A blocked page can still be listed by its URL alone, because the crawler was never allowed to fetch the page and read a noindex instruction. To keep a page out of results you need a robots meta tag or an X-Robots-Tag header, which means the page has to stay crawlable.
No. They are the only two special characters, section 2.2.3, and they only work in a path. A dot matches a dot, a plus matches a plus, and a $ anywhere but at the end matches a literal dollar sign.
This runs as client-side JavaScript. Keys, tokens, payloads, and other inputs never leave your device.