Technical SEO basics: crawlability, sitemaps and robots.txt
Make sure search engines can crawl and index your site: a clean structure, an XML sitemap, a sensible robots.txt, canonical URLs, and no accidental noindex. Technical SEO removes the blockers.
Great content cannot rank if search engines cannot find, crawl and index it. Technical SEO is the unglamorous foundation that makes sure nothing stands between your pages and the search index.
For most small sites it comes down to a handful of things done right.
How search engines process your site
Search engines discover URLs, crawl them, and decide whether to index them. Technical SEO ensures each step works: your pages are linked and listed, crawlers are not blocked, and nothing accidentally tells search engines to ignore your content.
Technical SEO clears the first three steps so content can reach the fourth.
| Item | What it does |
|---|---|
| XML sitemap | Lists your URLs for search engines |
| robots.txt | Guides what crawlers may access |
| Canonical tags | Point to the preferred version of a page |
| Indexability | No accidental noindex on key pages |
| Clean URLs | Readable, stable, logical addresses |
| HTTPS | Secure, trusted connections |
A basic technical checklist
- 1Submit an XML sitemapGenerate one and submit it in your search console so pages are found.
- 2Check robots.txtMake sure it is not blocking pages you want indexed.
- 3Verify indexabilityConfirm important pages are not set to noindex.
- 4Set canonical URLsPoint duplicate or similar pages to one preferred version.
- 5Serve over HTTPSSecure the site; it is a baseline trust and ranking signal.
- ✓Content cannot rank if it cannot be crawled and indexed.
- ✓An XML sitemap lists your URLs for search engines.
- ✓robots.txt guides crawlers — do not block yourself.
- ✓Canonical tags resolve duplicate-content confusion.
- ✓Check for stray noindex tags when pages will not rank.