Complete Syntax of Robots.txt file-
Robots.txt is a text file. It inform to the search engine which webpage,directory,folder & images you want to allow or disallow for crawling and indexing.
Allow indexing of everything
User-agent: *Disallow:
Disallow indexing of everything
User-agent: *Disallow: /
Disallow indexing of a specific folder
User-agent: *Disallow: /folder/
Block a webpage –
User-agent: *Disallow: /webpage.html
Block image folder-
User-agent: *Disallow: /images/
No comments:
Post a Comment