Crawlability and indexing get talked about as one problem, but they are two different failure points, and an SEO company has to diagnose which one it is dealing with before touching anything. Crawlability is whether Googlebot can physically reach and parse a page. Indexing is whether Google decides that page is worth storing and eligible to rank once it has been crawled. A page can fail at either stage for completely different reasons, and the fix for one can actively make the other worse if it is applied carelessly, which is exactly what happens when a site owner blocks a section in robots.txt to “clean up indexing” and instead creates a worse problem.
That last scenario is more common than it should be. Google’s own documentation is explicit that disallowing a URL in robots.txt stops crawling, not indexing. If other pages link to that blocked URL, Google can still index it using the link text and surrounding context, just without a title or description in the search result, the classic “no information is available for this page” listing. The only way to reliably keep a page out of the index is a noindex directive on a page Google is actually allowed to crawl. Mixing the two up is one of the more expensive technical mistakes a site can carry for months without anyone noticing, because Search Console will show the page as “Indexed, though blocked by robots.txt” rather than throwing an obvious error.
Reading Crawl Budget Correctly
Crawl budget is the combination of crawl capacity (how many connections Googlebot can make without straining the server) and crawl demand (how much of the site Google actually wants to check, based on popularity and how often content changes). Google’s own guidance is specific about when this is actually worth optimizing for: sites in the range of a million or more unique pages with content that changes roughly weekly, or sites around 10,000 pages with content changing daily, plus any site carrying a large volume of URLs sitting in “Discovered, currently not indexed” in Search Console. Most client sites, including a lot of mid-size B2B and service-business sites, simply do not have enough pages or change frequently enough for crawl budget to be the binding constraint. Content quality and internal linking usually are.
Where crawl budget genuinely matters, an SEO company is not guessing. The two real diagnostic sources are:
- Search Console’s Crawl Stats report (Settings > Crawl Stats), which breaks crawl requests down by response code, file type, crawl purpose (discovery vs. refresh), and Googlebot type, and shows the trend over time. The useful read here is not the raw request count, it is whether the discovery share is climbing while indexed-page count stays flat (a sign Google is spending requests on low-value new URLs, often faceted navigation or internal search results) or whether 5xx and timeout responses are spiking (a sign the server itself is the bottleneck, not the site’s structure).
- Server log file analysis, using a tool like Screaming Frog’s Log File Analyser or Botify, which reads the actual raw access logs to show exactly which URLs Googlebot requested, how often, and what it received. This is the only way to see the gap between what a crawler tool finds when it crawls the site and what Googlebot is genuinely spending its budget on, and it is standard practice on any site large enough for crawl budget to be a real concern.
For sites where crawl budget is in play, the practical levers are trimming crawlable space that has no ranking value (faceted navigation combinations, internal search result pages, tag archives with thin content), consolidating duplicate URLs created by parameters, and fixing redirect chains, all covered below.
Robots.txt: Getting the Front Door Right
Robots.txt is the first file Googlebot checks, and it is also where small syntax mistakes do disproportionate damage. A basic, correctly scoped file looks like this:
User-agent: *
Disallow: /cart/
Disallow: /search-results/
Allow: /search-results/category/
Sitemap: https://www.example.com/sitemap.xmlThe most common errors an SEO company finds during an audit:
- Accidental blanket disallows left over from a staging environment that never got removed before launch.
- Blocking CSS and JS directories, which prevents Google from rendering the page properly, not just crawling it.
- Wildcard rules that catch more URL patterns than intended, quietly disallowing pages nobody meant to hide.
According to the 2024 Web Almanac, HTTP Archive’s annual analysis of the web built on crawl data from millions of sites, robots.txt files themselves return a 404 status on 14.1% of mobile sites and 14.3% of desktop sites tested. A missing robots.txt is not catastrophic on its own (Google generally treats a 404 there as no crawl restrictions at all), but it usually signals a site that has never had its crawl configuration reviewed, and it is worth checking for what else was never set up.
XML Sitemaps That Earn Their Keep
A sitemap’s only job is to list the URLs the site actually wants indexed, in a format Google can fetch efficiently. Google’s sitemap guidance is direct that submitting one is a hint, not a guarantee Google will crawl every URL in it, which is why the quality of what goes into the file matters more than its existence. The practices that consistently improve outcomes:
- Include only canonical, indexable, 200-status URLs. Redirects, noindexed pages, and non-canonical duplicates in a sitemap send a mixed signal and waste the crawl requests Google does allocate to checking it.
- Keep
lastmoddates accurate. A sitemap where every URL shows the same date regardless of whether the page actually changed trains Google to discount the field entirely. - Split large sites into multiple sitemap files referenced by a sitemap index, since each individual sitemap file is capped at 50,000 URLs or 50MB uncompressed per Google’s sitemap size limits.
- Reference the sitemap location in robots.txt and submit it directly in Search Console so both paths confirm the same URL set.
For a site already struggling with crawl efficiency, a clean sitemap functions as a priority list, it does not replace the deeper structural fixes covered in our guide to site structure and architecture, but it does tell Google which URLs the site considers worth its time first.
Canonical Tags and Real Duplicate Content Control
A canonical tag tells Google which version of a URL to treat as the master copy when multiple URLs serve the same or substantially similar content, which is common on sites with filtered category pages, tracking parameters, or both HTTP and HTTPS or www and non-www variants still resolving. The syntax is simple:
<link rel="canonical" href="https://www.example.com/products/blue-widget" />The 2024 Web Almanac puts canonical tag adoption at 65% of mobile pages and 69% of desktop pages, up from 61% and 59% respectively in 2022, the first year the tag has been applied at roughly equal rates across mobile and desktop. That still leaves close to a third of pages on the open web shipping with no explicit canonical signal, which means Google is left to choose a canonical algorithmically using signals like internal linking patterns and redirects, and its choice does not always match the URL the business actually wants ranking. One correction worth making explicitly here: Google’s own documentation says not to use robots.txt for canonicalization. Of the three signals Google actually weighs, they are not equally strong:
| Canonicalization signal | Strength |
|---|---|
| 301 redirect to the preferred URL | Strongest |
| <!–INLINECODE1–> tag | Strong, but not absolute |
| Preferred URL listed only in the sitemap | Weakest |
An SEO company auditing canonical setup checks all three line up, not just whether a tag exists somewhere in the head.
Noindex and URL Parameter Cleanup, Done the Current Way
Noindex is actually a comparatively rare directive across the live web. The Web Almanac found the noindex meta robots rule applied on only about 3.9% of mobile pages and 4.7% of desktop pages in its 2024 dataset. In practice, the more common audit finding is not overuse, it is a noindex tag left on a page that should be ranking (often a leftover from staging, or applied site-wide to a template that later got real content), or a noindex tag sitting on a page that is also blocked in robots.txt, which makes it invisible to Google entirely since a crawler that cannot access the page also cannot see the noindex instruction telling it to drop the page from the index.
URL parameters (session IDs, sort orders, tracking tags like utm_source) used to be managed through Search Console’s URL Parameters tool, which Google deprecated and removed in April 2022 after finding that roughly 99% of the configurations site owners had set in it were not actually useful for crawling. There is no equivalent tool to point readers to today, and an SEO company should not reference it as if it still exists. The current approach is to let Google’s automatic parameter handling work for most cases, use a self-referencing canonical tag pointing parameterized URLs to the clean version when the parameter does not change the content meaningfully, and reserve robots.txt disallow rules for parameter patterns that are generating large volumes of low-value crawlable URLs (like infinite calendar pagination or internal search result pages), not for routine tracking parameters.
Using Google’s Indexing API the Right Way
This is the section most often gotten wrong in SEO content, including in plenty of agency blog posts. Google’s Indexing API is not a general tool for getting time-sensitive content like blog posts, news articles, or product launches crawled faster. Google’s documentation is explicit that the API is scoped to exactly two structured-data content types: pages marked up with JobPosting schema, and pages marked up with BroadcastEvent embedded in VideoObject schema for livestreams. The default quota is 200 URL notifications per day, and Google’s spam policies apply to anything submitted through it. Using the API outside its documented scope, which does happen when sites try to use it as a faster version of “request indexing,” risks the request being ignored at best and the account’s quota being throttled at worst.
For everything outside job postings and livestream events, the indexing levers an SEO company actually has are a clean, accurate sitemap, solid internal linking from already-indexed pages, and the manual “Request Indexing” option in Search Console’s URL Inspection tool for individual priority URLs. None of those are instant, but they are the documented, supported path, which the Indexing API is not for general content.
Fixing Redirect Chains and Loops
A redirect chain is a URL that redirects to a URL that redirects again before landing on the final destination (A to B to C to D). A redirect loop is a URL that eventually redirects back to itself. Both cost crawl budget every time Googlebot encounters them, since each hop is a separate request, and both dilute the ranking signals a redirect is supposed to pass through cleanly. The fix is always the same: every redirect should resolve in a single hop straight to the final URL, found by crawling the site with a tool like Screaming Frog and cross-referencing against the redirect rules in the CMS or server config.
A large share of chains trace back to old internal links that were never updated after a URL changed, a page links to a URL that has since been redirected, which then might redirect again from a later migration. That maintenance work, finding and repointing internal links at their final destination rather than at a redirect, is the part of “internal linking” that belongs to crawl efficiency specifically. The broader work of designing a site’s linking architecture and hierarchy from scratch is its own discipline, covered in our guide to site structure and architecture linked above, while where crawlability and indexing fit inside a full technical audit process is covered in our technical SEO audit guide. Mobile-first indexing also means Google evaluates the mobile version of a page first for crawling and indexing decisions by default, which is why the mobile and desktop figures cited throughout this guide are rarely identical, a topic covered in more depth in our guide to mobile SEO.
