@ | Posted 22 Sep. 2017
Configure this file robots.txt as soon as you start your domain. By default this file isn’t configured, you have to configure it. It contains only 2 commands, “allow or disallow”.
What options you can have on your website related to this file: * (means all)
Folder (any specific folder you want to disallow) Files (any specific file you want to disallow)
Images (any images you want to disallow)
Videos etc. (any videos you want to disallow)
Only in case of “DISALLOW OR DENY” you specify or list the above options. We allow it only when our on
page is totally complete. If nothing is mentioned in the file than everything is allowed.
Commands for File:
user-agent: * (this option is by default in the robot file)
Disallow:/ (this options means the website is out of SE and Google doesn’t has the right to access it yet,
it means all is disallowed)