How Does an SEO Company Improve Website Crawlability and Indexing?

An SEO company works in two passes. First it confirms that Google can reach and render the pages that matter: robots.txt rules, redirects, server errors and crawl statistics. Then it confirms that Google chooses to index them: noindex tags, canonicals, sitemaps, and duplicate or thin pages. Working out which pass is failing comes before any fix, because the fix for one can make the other worse.

Where to Start: Read the Page Indexing Report

Search Console’s Page indexing report shows which URLs Google knows about, which are indexed, and the reason each one is not. The reason labels point to the pass that is failing:

Reason	What it means	Which pass to fix
URL blocked by robots.txt	Google was not allowed to crawl the page	Crawling (configuration)
URL marked 'noindex'	Google crawled the page and found a noindex directive	Indexing (configuration)
Redirect error	A redirect chain was too long, looped, ran past the maximum URL length or hit a bad or empty URL	Crawling
Discovered, not yet indexed	Google knows the URL but has not crawled it yet	Crawling capacity or priority
Crawled, not indexed	Google fetched the page and chose not to index it	Indexing (check the selected canonical and rendering, then content and links)
Indexed, though blocked by robots.txt	The page was indexed without being crawled, from links elsewhere	Both: the robots rule is not doing what was intended

In the report, the two middle labels read Discovered - currently not indexed and Crawled - currently not indexed.

Google notes that a URL can be left out of the index for good reasons, such as an intentional robots.txt rule, a noindex tag or a duplicate, so the job is to check that each excluded URL is excluded on purpose. Google also says sites with fewer than 500 pages probably do not need this report and can check key pages with a site search instead.

When Google Crawls a Page but Does Not Index It

On a site Google can already crawl, the harder problem is selection. The crawled-but-not-indexed status means Google fetched the page and decided against storing it; Google adds that it “may or may not be indexed in the future” and that there is no need to resubmit it. Before treating it as a content problem, rule out the technical explanations in URL Inspection: whether Google chose a different URL as canonical, and whether the page renders its main content. What remains is editorial and structural:

• Consolidate thin or overlapping pages that answer the same query into one stronger page, with redirects from the old URLs.
• Improve the pages that should rank so they answer the query better than what Google already shows.
• Link to those pages from pages that are already indexed and receive traffic. How that link structure is planned belongs to how an SEO company optimizes website structure.

Robots.txt Controls Crawling, Not Indexing

Google’s robots.txt introduction says the file “is not a mechanism for keeping a web page out of Google.” If other pages link to a blocked URL, Google can still index it without visiting it, and the result appears without a description. To keep a page out of search, use a noindex directive on a page Google is allowed to crawl, or password-protect it. A noindex tag on a page that robots.txt also blocks is never seen, which is one way the “Indexed, though blocked by robots.txt” warning arises.

A basic, correctly scoped file:

User-agent: *
Disallow: /cart/
Disallow: /search-results/
Allow: /cart/help/

Sitemap: https://www.example.com/sitemap.xml

The errors an audit looks for: a blanket disallow left over from a staging site, blocked script or style files that the page needs (Google’s guide warns that blocking such resources means it “won’t do a good job of analyzing pages” that depend on them), and wildcard rules that catch more URLs than intended. The 2024 Web Almanac found robots.txt returned a 404 on 14.1% of mobile sites and 14.3% of desktop sites. A missing file blocks nothing: Google’s Crawl Stats help says a 404 for robots.txt means Google “can crawl any URLs on the site.”

Does Your Site Need Crawl Budget Work?

Unless the site is very large or changes very fast, no. Google’s crawl budget guide says it is written for very large and frequently updated sites: roughly 1 million or more unique pages changing about weekly, 10,000 or more pages changing daily, or sites with a large share of URLs that Google has discovered but not yet indexed. Google calls those numbers “a rough estimate” and does not present them as exact thresholds. For other sites, it says keeping the sitemap current and checking the Page indexing report regularly “is adequate.”

Where crawl budget does apply, two sources replace guessing. Search Console’s Crawl Stats report breaks crawl requests down by response code, file type, purpose (discovery or refresh) and Googlebot type over time. Server log analysis shows which URLs Googlebot requested and what it received. Warning signs are discovery requests climbing while indexed pages stay flat, which points to low-value URLs such as filter combinations or internal search results, and a rise in server errors, which points to the server itself.

XML Sitemaps: A List of Pages You Want Indexed

Google’s sitemap guidance calls submitting a sitemap “merely a hint”; what goes in it matters more than whether it exists:

• List only canonical, indexable URLs that return 200. Redirects, noindexed pages and duplicates send mixed signals.
• Keep lastmod accurate. Google uses it only if it is “consistently and verifiably” accurate, and it ignores priority and changefreq.
• Split large sites: a single sitemap is limited to 50,000 URLs or 50MB uncompressed, so larger sites use several files and a sitemap index.
• Reference the sitemap in robots.txt and submit it in Search Console.

Canonical Tags and Duplicate URLs

A canonical tells Google which URL to treat as the main version when several serve the same or very similar content, such as tracking parameters, filtered pages, or HTTP and HTTPS versions:

<link rel="canonical" href="https://www.example.com/products/blue-widget" />

The tag goes on the duplicate URLs, pointing to the clean version, and the clean URL carries a self-referencing canonical pointing to itself. Google’s canonicalization guide lists the methods in order of strength:

Method	Strength, in Google's wording
Redirect to the preferred URL	Strong signal
rel=canonical link annotation	Strong signal
Inclusion in the sitemap	Weak signal

Google adds that the methods stack when combined, warns against using robots.txt for canonicalization, and says that a site without any canonical preference will “likely do just fine” because Google picks a version itself. The 2024 Web Almanac puts canonical tag use at 65% of mobile pages and 69% of desktop pages, up from 61% and 59% in 2022.

Noindex and URL Parameters

The Web Almanac found noindex on 3.9% of mobile pages and 4.7% of desktop pages in 2024. The more useful audit check is the reverse of overuse: a noindex left on a page that should rank, carried over from staging or set on a template that later received finished content. Check it in URL Inspection’s live test, which shows whether indexing is allowed.

Search Console’s URL Parameters tool no longer exists. Google announced its deprecation on March 28, 2022, saying only about 1% of the parameter configurations in it were useful for crawling and that its crawlers learn how to handle parameters automatically. Use canonical tags for parameters that do not change the content, and robots.txt rules only for parameter patterns that generate large numbers of low-value URLs, such as internal search results or endless calendar pages.

What the Indexing API Can and Cannot Do

Google’s Indexing API documentation limits the API to pages with JobPosting structured data or BroadcastEvent embedded in a VideoObject, that is, job postings and livestreams. Google’s spam policies apply to anything submitted, and the default quota is 200 publish requests per day per project. For blog posts, product pages and everything else, the supported routes are an accurate sitemap, internal links from indexed pages, and Request Indexing in URL Inspection for individual priority URLs.

Redirect Chains and Mobile-First Checks

Our standard is that every redirect resolves in one hop to the final URL. A short chain still works, but each hop adds a request, and chains that run too long or loop show up in the Page indexing report as redirect errors and in a crawler’s redirect reports; the repair is to update the redirect rules and repoint internal links straight to the final URL.

Google completed mobile-first indexing on October 31, 2023, so the mobile version of a page is the one it crawls and indexes. A mobile site blocked by robots.txt while the desktop site is allowed is one of the failure types Google named in that announcement. For device-specific checks, go to how an SEO company optimizes websites for mobile SEO. One caution when reading published statistics: Web Almanac mobile and desktop figures differ because HTTP Archive runs separate mobile and desktop crawls, not because of how Google indexes.

What You Should Get From the Agency

Where crawling and indexing sit inside a full audit is explained in how an SEO company conducts a technical SEO audit. For this part of the work, ask for a list of every important URL, built from the CMS, the XML sitemaps and a crawl, with each excluded URL sorted into one of three buckets: configuration (robots.txt, noindex, redirects), selection (crawled or discovered but not indexed) or intentional. The Page indexing report cannot supply that list on its own: its help page says the example URLs are limited to 1,000 and are not guaranteed to show every URL in a given status. Configuration problems can be fixed directly once found; selection problems start with URL Inspection on representative pages before any content or link work, and the list should give a date to recheck the report.