Slide 22 of 39
How Does an SEO Company Conduct a Technical SEO Audit?
An SEO company runs a technical audit in passes: first anything that stops Google from crawling or indexing pages, then server responses and crawl behavior, then speed and Core Web Vitals, then site architecture, and finally mobile usability and HTTPS. The output is a prioritized fix list in which each finding is scored by how many URLs it affects, how much traffic those URLs carry and how severe it is.
What the Agency Needs Before It Starts
A useful audit needs access, not just a crawler. Expect to grant the agency access to Search Console and Google Analytics, to the CMS or a staging copy of the site, and, for large sites, to server or CDN logs. Without Search Console it cannot see which pages Google has excluded from the index; without logs it cannot see every URL Googlebot requested.
Pass 1: Crawling and Indexing Blockers
The audit starts with a full crawl in a tool such as Screaming Frog or Sitebulb, mapping every URL it can reach through links and flagging what keeps Google from finding or indexing pages: stray noindex tags left from a staging site, redirect chains and loops, and duplicate URLs without a canonical tag. Orphaned pages, which no internal link reaches, do not show up in a crawl that only follows links, so the crawler is also given the URLs listed in the XML sitemap, Search Console and analytics to compare against. Because Google indexes the mobile version of each page, the crawl runs with a smartphone user agent and is compared with a desktop crawl. Content, internal links or structured data that exist only on desktop are logged here as indexing problems.
Two files are checked in the same pass:
- robots.txt. In the Lighthouse audits the 2024 Web Almanac ran alongside its own data collection, 8.43% of desktop pages and 7.40% of mobile pages failed the check for a valid robots.txt file. Google does not require the file: its Crawl Stats help counts a 403, 404 or 410 for robots.txt as a successful response, but after a 429 or 5xx response Google might stop crawling the site. A misplaced disallow rule can stop Google from crawling a whole section without any visible error. It does not remove pages from the index, though: Google’s robots.txt introduction says it “is not a mechanism for keeping a web page out of Google”. The same page explains that a blocked URL can still be indexed if it is linked from elsewhere. Pages that must leave the index need a noindex tag and must stay crawlable. The agency runs the crawler with the live robots.txt rules to list every URL each rule blocks, then spot-checks key URLs in Search Console.
- The XML sitemap. Every URL listed should be indexable and return a 200 status, and every important indexable page should be listed. The sitemap helps Google discover URLs; an error in it does not block indexing on its own.
On canonical tags, the same Web Almanac chapter reports that 65% of mobile pages and 69% of desktop pages carry one. A missing tag matters only where duplicate URLs exist (parameters, filters, tracking codes, print versions), so the audit checks those URL patterns instead of flagging every page without a tag. Pages that depend on JavaScript get a render check: a crawl with rendering turned on, compared with the raw HTML, plus URL Inspection’s live test on key templates. Sites in several languages or countries also get an hreflang check; the setup is part of how an SEO company handles international SEO campaigns.
Pass 2: Server Responses and Crawl Behavior
The crawl and Search Console show response codes across the site: 404s that still receive links or traffic, 5xx errors that come and go, and redirect chains that waste crawling even when they resolve. Server logs show what Googlebot does: which pages it crawls, which it skips and where its requests go.
Log analysis earns its cost on large sites. Google’s crawl budget guide is written for very large and frequently updated sites, giving rough examples of about 1 million or more unique pages that change weekly, or 10,000 or more that change daily, and for sites where Search Console reports many URLs as discovered but not yet indexed. For smaller sites, Google says keeping the sitemap up to date and checking the Page indexing report regularly is adequate. The crawl budget and indexing fixes that follow from this pass are detailed in how an SEO company improves crawlability and indexing.
Pass 3: Speed and Core Web Vitals
The audit checks key templates against Google’s “good” thresholds for Largest Contentful Paint, Interaction to Next Paint and Cumulative Layout Shift. PageSpeed Insights shows two kinds of data: field data from Chrome users over the previous 28 days, which falls back to the whole origin when a URL has too little traffic, and a Lighthouse lab test of the single URL. The field data confirms that a problem exists; the lab test helps find its cause. What drives each metric and how the fixes are made is the subject of how an SEO company improves website user experience for SEO.
Pass 4: Site Architecture
The crawl data shows how pages connect: click depth from the home page, sections with few internal links, and important pages buried several levels down. The audit flags these with the affected URLs; the restructuring itself belongs to how an SEO company optimizes website structure.
Pass 5: Mobile Usability and HTTPS
The last pass covers device checks that the parity crawl does not: tap targets, intrusive pop-ups and mobile loading on physical phones, following the mobile checklist in how an SEO company optimizes websites for mobile SEO. It closes by confirming HTTPS sitewide: a valid certificate that is not close to expiry, HTTP-to-HTTPS redirects on every subdomain, and no mixed-content warnings. This is a pass-fail confirmation that needs revisiting after a migration or a platform change.
Checking Order Is Not Fixing Order
The passes above are the order of checking. Fixes are ordered by a score: how many URLs a finding affects, how much traffic or revenue those URLs carry, and how severe it is (blocks indexing, degrades performance, or is cosmetic). A noindex tag on a sitewide template goes to the top of the list; the same tag on three retired URLs may not make the first sprint.
What the Audit Should Deliver
A usable audit is a table with these columns:
| Column | What it holds |
|---|---|
| Issue | What is wrong, in one line |
| Affected URLs | The count, with examples |
| Evidence | Crawl data, Search Console report, logs or field data |
| Impact | Blocks indexing, degrades performance, or cosmetic |
| Fix | The change to make |
| Owner | Agency, your developer or your content team |
| Effort | A rough estimate in hours or days |
Before You Sign Off on an Audit
Check three things: every finding lists the affected URLs and the evidence behind it, every item has an owner and an effort estimate, and the first sprint starts with whatever blocks indexing on your highest-traffic templates. An audit that is a raw export of every crawler warning, with no scoring, leaves the prioritization to you. After fixes ship, the agency should re-crawl the affected URLs and check key ones in URL Inspection, so the list records what was fixed and confirmed, not only what was recommended.