SEO website

Why robots.txt is Important for SEO while making website ?


Urban Desi
@ | Posted 22 Sep. 2017


Initially when the website is under construction, you put a copy past content on the website. It is used for the finalization of website design from the client. When you are copying and pasting the content always remember to give a message to the Search Engine SE that don’t check my website at the moment, if you haven’t done that then SE will penalize you.

For giving out a message to Google you use or edit a file robots.txt (only for the Search Engine ) This file is available in every hosting or you can make it in public_html. Whenever you will make a website, the above mentioned file will already be in it but will be empty. Whenever Google goes through your website the first thing it checks is this file and if he sees that you have allowed Google to check it then it won’t add it in its database and the website will be closed even before doing its SEO and when google sees that you have denied access to the website then it will go back without checking.

You can allow or deny robots to crawl the pages.