robots.txt
Publish https://yourdomain.com/robots.txt. Allow important marketing and product paths. Disallow only what must stay private (admin, carts, internal search).
Include a Sitemap: line pointing at your XML sitemap. Re-check after every platform or WAF change — Attack Mode and aggressive bot rules can break crawlers and auditors alike.
XML sitemap
List canonical HTTPS URLs that return 200. Keep lastmod honest. Split large sites into a sitemap index.
Submit the sitemap in Google Search Console and Bing Webmaster Tools. Remove dead URLs promptly so crawl budget is not wasted.
Canonical tags
Every indexable page should declare a preferred absolute HTTPS URL with <link rel="canonical" href="…">.
Avoid pointing many distinct pages at a single catch-all canonical. Prefer self-canonicals on primary templates and consistent trailing-slash / www policy.
Verify in an audit
A good website audit checks robots reachability, sitemap presence, and canonical presence on the measured URL — then tells you exactly what is missing. That is part of every AuditHQ website lab run.