Setup robots.txt

From the beginning I made some mistakes in the indexing of SEO Hero site, but corrected.

Now indexing site happens according to plan.

I created a site map. Read more at the link — seoheronews.com/sitemap.

However, I did not realize the response codes of the server and did not close all the technical pages from the Google and Yahoo. Now I created a file robots.txt and closed from indexing irrelevant pages.

Some pages closed from indexing via a special meta tag robots.

User-agent: *
Disallow: /cp/
Disallow: /search/
Disallow: /goto?
Disallow: /login
Disallow: /api
Host: https://seoheronews.com
Sitemap: https://seoheronews.com/sitemap.xml