So its been awhile since we did a blog entry, Mainly because we have been studying SEO and taking full advantage of your robots.txt to help get better results.

 It came to our attention because we had all the SEF turned on, Google still was indexing none SEF urls.

 We had 758 pages indexed. Now every day we are seeing pages vanish (thats our goal)

 First off if you didnt know, Install xmap it generated a sitemap not only for your site but it gives you an xml built on the fly that can be submitted to google, yahoo, ask and msn and several other sites. 

 You can also add this to your robots.txt

Sitemap: http://www.joomlamafia.com/index.php?option=com_xmap&sitemap=1&view=xml&no_html=1

 Whats this do? Every time a bot comes to your site it looks at robots.txt to know what not to index and what to index... It just found out first thing is all bots this applies to, heres my sitemap, heres all my pages!!! But if any of these links are present (which we know they are not because this is what all our viewers of our site get to see when they click sitemap link)

 Next thing we added was

Disallow: index2.php

 and then

Disallow: /component/content/article/* Disallow: /component/search/* 

Disallow: /component/*

This keeps any component pages from getting indexed, as well as any search pages and the firt keeps the articles in none sef format from being indexed and causing double content.

How Can I Apply This To My Site?