Robots file for search engine ?

If you want to make the search engines to ignore a folder in your WWW directory, you have to make robot.txt file in your WWW directory and give it the following code :

User-agent: *
Disallow: /beta*/
Disallow: /wp-admin/
Disallow: /wp-includes/

note that : the above example for the wordpress website but it  work the same way to any website

 

Advertisements
Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: