Indexing Dashboard
The Indexing Dashboard gives you a complete view of how Google indexes your website. It uses the Google URL Inspection API to check each page individually, tracks status changes over time, and shows you exactly which pages are indexed, which aren't, and why.
The dashboard has three tabs:
Overview — High-level indexing status, trends, and recent movements
Visual Diagnostics — Charts that expose structural site issues (velocity, directory health, crawl budget, internal links, funnel)
Log Analyzer — A combined data table merging GSC data, internal links, and server log hits with actionable insights
Getting Started
To use the Indexing Dashboard, you need a Google Search Console property connected to SEO Utils with at least one sitemap added.
Connect a GSC Property
If you haven't already, connect your Google Search Console property using either a Google OAuth Token or a Google Service Account. See the Google Search Console setup guide for instructions.
Add a Sitemap (if needed)
When you connect a GSC property, SEO Utils automatically detects and adds sitemaps from Google Search Console. If sitemaps were found, you can skip this step.
If no sitemaps were detected, or you want to add a different sitemap, navigate to Sitemaps from the property dropdown and add your sitemap URL (e.g., https://yoursite.com/sitemap.xml). SEO Utils will automatically fetch URLs from your sitemap daily.
Open the Indexing Dashboard
Select your property from the dropdown, then click Indexing in the navigation menu. The dashboard will begin collecting data automatically.
On first visit, the dashboard shows "Based on 0/N URLs inspected" because no URLs have been inspected yet. Click Run Initial Scan in the top-right to start the first inspection batch, or wait for the automatic inspection to run (every 4 hours).
Shared Controls
These controls appear at the top of the dashboard and apply across all three tabs:
Date Filter
The date dropdown in the top-right controls the time range for the chart, table impressions/clicks, movements, funnel, and velocity data.
Available presets: 7 days, 14 days, 30 days, 2 months, 3 months, 6 months, 12 months. The default is 30 days.
Sitemap Errors Banner
If any sitemap has a fetch error (e.g., HTTP 404, timeout), a dismissible banner appears at the top showing the errors. This is visible on all tabs.
Settings
Click the cog icon in the top-right to open the Indexing Settings modal. See the Settings section below for details.
Tab 1: Overview
The Overview tab shows your high-level indexing status, historical trends, and recent status changes.
Status Tabs & Chart
Three tabs at the top — All, Indexed, Not indexed — filter both the chart and pages table. Use the Show % toggle to switch between counts and percentages.
When you select Not indexed, clickable sub-status chips appear (e.g., "Crawled - currently not indexed", "Not found (404)") to drill down by reason. Select multiple chips to combine filters.
The Indexing Overview Chart shows a stacked bar trend over time — green for indexed, orange/multi-color for not indexed sub-statuses.
Pages Table
Shows every URL from your sitemap with columns for URL, Clicks, Impressions, Status (coverage state badge), Last Crawl, Rich Results, and Last Inspection.
Filters: Page URL search, Status, Content Group, and "Pages at risk of de-indexing" (indexed pages not crawled in 90+ days). Click Advanced for Clicks/Impressions range filters and Last Crawl date picker.
Actions: Click ... on any row to Check Index (re-inspect via API) or Request Indexing (submit to Google). Select multiple rows for bulk Check Index or Submit Index.
Recent Movements
Tracks indexing status changes over time across three tabs:
All Movements — Every status change with a Type column (Coverage, Indexing, Robots, Canonical)
Indexing Changes — Only coverage and indexing state changes
Recently Published & Not Indexed — New URLs still not indexed within the date range
Tab 2: Visual Diagnostics
The Visual Diagnostics tab shows five charts that help you identify structural indexing problems at a glance. Data is loaded only when you navigate to this tab.
Internal Link Crawler
At the top of the Diagnostics tab, a Crawler Progress Card shows the status of the internal link crawler:
Not run yet — "Internal link crawler has not been run yet." with a Run Link Crawler button
Running — "Crawling... X/Y URLs" with live progress updates
Completed — "Last crawl: Apr 5, 2026 — 2000 URLs"
Click Run Link Crawler to start a manual crawl. The crawler uses headless Chrome to visit each sitemap URL, extract all internal <a> links, and store the link graph (anchor text, rel attributes, follow/nofollow). This data powers the scatter plot and the inlink counts in the Log Analyzer table.
The crawler processes up to 2,000 URLs per cycle, prioritizing uncrawled URLs first. For larger sites, the full site is covered over multiple weekly cycles. You can configure the crawl rate and schedule in Settings.
Indexing Velocity
This chart answers: "Is Google getting faster or slower at indexing my new content?"
Blue line — Median (P50) time-to-index in days
Red shaded area — 90th percentile (P90) band, showing how long the slowest 10% of URLs take
The X-axis shows weeks. Each data point represents all URLs published that week and how long they took to reach "Submitted and indexed" status.
This is a forward-looking metric. It only tracks URLs discovered after you set up SEO Utils, so the data won't include historical pages that existed before your first sitemap sync. New users will see an empty state: "Awaiting new content. Velocity tracking begins when you publish new URLs to your sitemap."
Directory Health
Horizontal stacked bars showing the indexed vs. not-indexed ratio for each subdirectory of your site.
Green = indexed percentage
Red = not-indexed percentage
Shows the top 10 directories by URL count, with an "Other" bucket for the rest
Root-level pages (like
/about,/contact) are grouped into the/bucket
Use the Group By dropdown (top-right of the chart) to switch between:
Directory Path (Auto) — Groups by first URL path segment (default)
Content Groups — Groups by the content groups you've created in the Insights tab (using URL filters or manual page selection)
Crawl Budget by Directory
A bar chart showing how Googlebot allocates its crawl budget across your site's directories.
This chart is only available when you have a log report linked to this domain. If no log report is linked, you'll see: "Link a log report to see crawl budget allocation by directory."
The "aha moment": Compare the Directory Health chart with the Crawl Budget chart side by side. If /search/ is only 15% indexed but consumes 75% of all Googlebot hits, you've found a crawl trap that needs fixing.
Internal Links vs. Indexation
A scatter plot that shows the correlation between internal link count and indexation success.
X-axis — Number of internal inlinks (from the local crawler)
Y-axis — Two bands: "Indexed" (top) and "Not Indexed" (bottom)
Green dots = indexed URLs, Red dots = not-indexed URLs
Dot sizes are uniform (no bubble chart)
You'll typically see orphan pages (low inlinks) clustered in the "Not Indexed" band, visually proving that pages need internal links to get indexed.
When log data is available, a Hide Zero-Hit Pages toggle appears to filter out URLs that Googlebot hasn't visited.
This chart requires the internal link crawler to have run. If the crawler hasn't been run yet, you'll see: "Run the internal link crawler to see this chart."
Indexing Funnel
A horizontal bar chart showing where your URLs currently sit in the indexing pipeline. Each bar's width is proportional to the total sitemap count, so you can instantly see the distribution.
In Sitemap
Gray
Total URLs from your sitemap (baseline — always 100%)
Discovered
Amber
URLs Google knows about but hasn't crawled yet — may need better internal linking
Crawled
Red
URLs Google crawled but rejected — content quality or technical issues
Indexed
Green
Successfully indexed and eligible to appear in search
Impressions
Violet
Indexed pages that received at least 1 impression in the selected period
Each bar shows the count and percentage of total. Hover over any bar for a description of what the stage means.
Intentional Exclusions are listed below the bars as a text summary:
Blocked by robots.txt — URLs your robots.txt prevents Google from crawling
Excluded (noindex / duplicate canonical) — URLs intentionally excluded from the index
A large red "Crawled" bar relative to the green "Indexed" bar means Google is visiting your pages but rejecting the content. This is usually the most actionable insight — check those pages for thin content, duplicate issues, or soft 404s.
Tab 3: Log Analyzer
The Log Analyzer tab combines data from three sources into a single actionable table:
GSC URL Inspection API — Coverage state for each URL
Internal Link Crawler — Inlink count per URL
Server Log Files — Googlebot hit counts from your server logs
Columns
URL
Any
The page path (click to open in browser)
GSC Status
URL Inspection API
Coverage state badge. Shows "Not in Sitemap" for URLs found only in logs
Inlinks
Local Crawler
Internal link count. Shows "—" if crawler hasn't run
Log Hits (30d)
Server Logs
Googlebot hit count. Shows "—" if no log report linked
Actionable Insight
Computed
Color-coded badge based on the rules below
Actionable Insights
Each URL gets an automatically computed insight based on its data:
Healthy
Green
URL is indexed
High Crawl Waste
Red
High log hits + not indexed (Googlebot keeps visiting but Google won't index)
Low Crawl Waste
Orange
Moderate log hits + not indexed
Orphan Page
Amber
Few or zero internal links + not indexed (needs internal links)
Not in Sitemap
Purple
URL found in server logs but not in your sitemap
The thresholds for "high" and "low" crawl waste and the orphan inlink threshold are configurable in Settings.
Filters
URL search — Text search across all URLs
GSC Status — Faceted filter (select one or more coverage states, or "Not in Sitemap")
Insight — Faceted filter (Healthy, High Crawl Waste, Orphan, etc.)
Inlinks — Range filter (from/to)
Log Hits — Range filter (from/to)
CSV Export
Click the Export CSV button to download the full merged dataset as a CSV file. The export includes all columns and all rows (not just the current page).
The Log Analyzer table works with whatever data is available. If you haven't run the crawler, the Inlinks column shows "—". If no log report is linked, the Log Hits column shows "—". The table is still useful with just GSC data alone.
Settings
Click the cog icon in the top-right to open the Indexing Settings modal. The settings are organized into four sections.
Auto Submit for Indexing
Auto Submit for Indexing
When enabled, automatically submits not-indexed URLs to Google via the Indexing API. URL inspection always runs regardless of this setting.
Resubmission Cooldown
Minimum days between resubmission attempts for the same URL. Default: 7 days. Only shown when auto-submit is on.
URL Normalization
These settings control how URLs are normalized for matching across GSC data, server logs, and crawler results.
Trailing Slash
How to handle trailing slashes: Follow Sitemap (default — strips trailing slashes), Always Add, or Always Strip
Keep Parameters
Query parameters to preserve (e.g., p, page, id, lang). Type a parameter name and press Enter to add it as a tag. All other parameters (including UTM tags) are stripped.
The Recalculate Normalized Paths button recomputes all normalized paths when you change these settings. This affects how URLs are joined across the three data sources.
Actionable Insight Thresholds
Configure the sensitivity of the Log Analyzer's insight engine:
High Crawl Waste (hits >)
20
Log hits above this trigger "High Crawl Waste" for not-indexed pages
Low Crawl Waste (hits >)
5
Log hits above this trigger "Low Crawl Waste" for not-indexed pages
Orphan Page (inlinks <)
2
Pages with fewer inlinks than this are flagged as orphan pages
Internal Link Crawler Settings
Requests per Second
1
How fast the crawler visits your pages. Keep low to avoid overwhelming your server.
Max Chrome Tabs
5
Number of concurrent headless Chrome tabs. Each uses ~50-100MB RAM.
Auto-Crawl Schedule
Weekly
How often the crawler runs automatically: Weekly, Bi-weekly, Monthly, or Manual Only
The crawler uses headless Chrome to render pages (capturing JavaScript-rendered links). At the default rate of 1 request/second with 5 tabs, it processes ~300 pages/minute. A 2,000-URL batch takes about 7 minutes.
How It Works Behind the Scenes
URL Inspection
SEO Utils automatically inspects your URLs using the Google URL Inspection API every 4 hours. URLs are prioritized in this order:
Never inspected — New URLs from sitemap
Content updated — URLs where the sitemap lastmod changed
Not indexed — URLs that aren't indexed yet
Indexed — Re-checked on a rotating basis (lowest priority)
Google enforces a limit of 2,000 URL Inspection API requests per day per property. For sites with more URLs, SEO Utils automatically spreads inspections across multiple days.
URL Normalization Pipeline
To join data from GSC (full URLs), server logs (request paths only), and the crawler (full URLs), all URLs pass through a normalization pipeline:
Strip protocol and hostname →
/blog/postStandardize trailing slashes (based on your setting)
Remove tracking parameters (UTM, gclid, fbclid, etc.)
Keep only configured structural parameters
Lowercase the path
This produces a normalized_path used for exact-match joins across all three data sources.
Googlebot Verification
When importing server logs, SEO Utils verifies Googlebot requests using Google's officially published IP ranges (CIDR blocks). This prevents spoofed Googlebot user agents from polluting your data. IP ranges are refreshed daily from Google's public endpoint.
Troubleshooting
Chart shows "No chart data available"
The chart requires at least one inspection batch to complete. Either:
Click Run Initial Scan in the top-right to start an immediate inspection
Wait for the automatic inspection cycle (runs every 4 hours)
Chart data is generated from inspection results, so URLs must be inspected before the chart has data to display.
Dashboard shows "Based on 0/N URLs inspected"
This means URLs have been discovered from your sitemap but haven't been inspected yet. This is normal on first setup. Click Run Initial Scan or wait for the automatic cycle.
"No sitemaps found" error
Your property doesn't have a sitemap configured in SEO Utils. Navigate to Sitemaps from the property navigation and add your sitemap URL.
Velocity chart is empty
The velocity chart only tracks URLs discovered after you set up SEO Utils. If all your URLs were already in the sitemap when you first connected, the chart has no data to show. Publish new content to your sitemap and the chart will populate as those URLs get indexed.
Scatter plot says "Run the internal link crawler"
The Internal Links vs. Indexation chart requires crawl data. Go to the Visual Diagnostics tab and click Run Link Crawler at the top. The scatter plot will populate once the crawl completes.
Crawl Budget chart says "Link a log report"
The Crawl Budget chart requires server log data. Create a log report in the Log Analyzer section of the sidebar, import your server logs, and make sure the report's domain matches your GSC property domain.
Log Analyzer columns show "—"
A dash means no data is available from that source:
Inlinks showing "—" → Run the internal link crawler
Log Hits showing "—" → Link a log report with server log data for this domain
The table still works with partial data — you don't need all three sources to use it.
Inspection seems slow
Each URL requires a network call to Google's API. Large sites with hundreds or thousands of URLs may take several minutes to complete a full inspection cycle. This is normal.
Quota exceeded — inspections stopped
Google limits URL Inspection API to 2,000 requests per day per property. When the quota is reached, remaining URLs are scheduled for the next day. The dashboard shows partial data based on what was inspected.
Clicks and Impressions show 0 for all URLs
Clicks and impressions come from Google Search Console performance data, which requires the GSC data sync to be running. Make sure your property has synced data (check the Performance page). The indexing dashboard joins with the performance data for the selected date range.
Last updated