You finish your sitemap, upload sitemap.xml, and then stop one step too early: crawlers can read the file only if they can find it. A clean robots.txt file with a proper Sitemap: line is the small bit of plumbing that keeps discovery boring in the best possible way.
This guide shows how to use a robots.txt generator to point search engines to your sitemap, what the file should contain, and how to avoid the classic mistake: accidentally blocking the same URLs you want indexed.
What robots.txt does, and what it does not do
robots.txt is a plain text file that lives at the root of your domain:
https://example.com/robots.txtIts main job is to give crawler instructions. You can allow broad crawling, block low-value paths, or point crawlers toward your sitemap. It is read before many bots request other URLs, so a broken file can create real SEO headaches.
But it is not a security feature. A Disallow rule asks compliant crawlers not to request a path; it does not hide private content from people, browsers, or bad bots. If a page is confidential, protect it with authentication, remove it from public URLs, or return the right server status. Do not rely on robots.txt as a lock.
The Sitemap line: the part most sites actually need
For many small sites, the most useful robots.txt content is simple:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThe Sitemap: line tells crawlers where your XML sitemap lives. Use the full absolute URL, including https://. It can sit anywhere in the file and does not need to be inside a User-agent block.
That single line helps Google, Bing, and other crawlers discover your URL list without waiting for every page to be found through links. You can still submit the sitemap in Google Search Console. In practice, using both is the stronger setup: robots.txt for automatic discovery, Search Console for status and error reporting.
How to generate robots.txt from your sitemap
The clean workflow is to build the sitemap and the matching robots.txt snippet together.
- Collect your canonical URLs. Include only the pages you want indexed. Avoid duplicate tracking URLs, staging pages, admin routes, and search-result pages.
- Generate the sitemap. Paste the URLs into the free NasrTech Sitemap Generator. It creates
sitemap.xml, an HTML sitemap, and arobots.txtsnippet in your browser, with no upload or sign-up. - Set the sitemap location. Use the final public URL, usually
https://example.com/sitemap.xml. - Copy the robots.txt output. Keep it short unless you have real paths to block.
- Upload both files to your site root. Your files should be reachable at
/sitemap.xmland/robots.txt. - Test the URLs. Open both in a browser. If either returns a login page, redirect loop, 404, or HTML error page, fix that before submitting.
If you are starting from zero, read the broader guide on how to create a sitemap first. If the sitemap is already live, the robots step is quick.
Safe defaults for most websites
Most websites do not need a long robots.txt. Long files often come from copying someone else's SEO setup without understanding it.
Here is a safe default:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlUse this when your public pages should be crawlable and your sitemap contains only canonical URLs. It is boring, readable, and hard to misinterpret.
Add Disallow rules only when you have a reason. Common examples include:
| Path | Why you might block it |
|---|---|
/admin/ | Back-office routes should not be crawled |
/cart/ | Personal shopping flows create crawl waste |
/search/ | Internal search pages can produce thin duplicates |
/?sort= | Sorting/filter URLs can multiply near-duplicate pages |
Blocking a path does not remove it from Google's index if it is already known. For removal or noindex decisions, you need page-level controls and proper HTTP behavior. robots.txt controls crawling, not every indexing outcome.
Mistakes that quietly break SEO
The dangerous line is usually this one:
Disallow: /It tells crawlers not to request anything under the domain. That can be useful for a private staging site. On production, it can cut off your entire public site. Before publishing a generated file, scan for that line and make sure it is truly intentional.
Also watch for these:
- Blocking the sitemap file itself. If you disallow the folder where
sitemap.xmllives, crawlers may not fetch it. - Using a relative Sitemap URL. Write
Sitemap: https://example.com/sitemap.xml, not just/sitemap.xml. - Pointing to the wrong domain. This happens after migrations from
example.nettoexample.com, or from non-www to www. - Listing URLs you do not want indexed. A sitemap is a recommendation list. Keep it clean.
- Forgetting HTTPS. If your site is HTTPS, your sitemap and
robots.txtreferences should be HTTPS too.
For the next step after publishing, use the guide on how to submit a sitemap to Google. It explains what "Success" really means inside Search Console.
A quick pre-publish checklist
Before you upload robots.txt, run this checklist:
- Can you open
https://yourdomain.com/robots.txtwithout logging in? - Can you open the exact sitemap URL listed in the file?
- Does the sitemap return XML, not a pretty 404 page?
- Does the file avoid accidental
Disallow: /on production? - Are all Sitemap URLs absolute and HTTPS?
- Did you include only canonical, indexable pages in the sitemap?
If all six are true, your crawler discovery setup is probably fine.
Frequently asked questions
Do I need robots.txt if I already submitted my sitemap to Google?
Yes, it is still useful. Search Console submission gives Google a direct path and reporting. The robots.txt Sitemap line gives crawlers an automatic discovery hint, including search engines where you have not created an account.
Can robots.txt force Google to index my pages?
No. A sitemap helps discovery, and robots.txt can point to that sitemap, but indexing still depends on content quality, accessibility, canonical signals, internal links, and crawler decisions.
Should I block duplicate pages in robots.txt?
Sometimes, but be careful. For many duplicate or filtered pages, canonical tags, noindex rules, or parameter handling may be a better fit. Blocking crawling can prevent crawlers from seeing page-level signals.
Is the NasrTech tool private?
Yes. The Sitemap Generator runs in your browser. Your URL list is not uploaded, and the tool formats the sitemap, HTML sitemap, and robots.txt snippet locally.
The bottom line
A good robots.txt file is usually short. Let crawlers access the public site, point them to sitemap.xml, and avoid dramatic rules unless you know why they exist. Use a robots.txt generator to reduce formatting mistakes, then test the two public URLs before submitting anything. Small file, big consequences, very worth getting right.
