Setup robots.txt

30546 site views

From the beginning I made some mistakes in the indexing of SEO Hero site, but corrected.

Now indexing site happens according to plan.

I created a site map. Read more at the link —

However, I did not realize the response codes of the server and did not close all the technical pages from the Google and Yahoo. Now I created a file robots.txt and closed from indexing irrelevant pages.

seo hero robotstxt

Some pages closed from indexing via a special meta tag robots.

User-agent: *
Disallow: /cp/
Disallow: /search/
Disallow: /goto?
Disallow: /login
Disallow: /api
set robots txt
Site views