A robots.txt file tells search engines what files and directories you don't want accessed or crawled on your site. Every Jive-powered community has this file, and it's stored in the root directory. Example: https://domainname.com/robots.txt.
If set up properly, a robots.txt file can be a useful tool. Here are a few tips to help you get started:
Optimize your file. The best thing you can do is keep your file simple! Only block accessible pages you don't wanted crawled. Here's an example of the an ideal robots.txt file:
If you're not sure which files to block, you can work with your company's developer team or your Jive support team.
Include your sitemap. If you're using Jive 7, you're in a good place. The core product includes automated sitemaps, which means newer, fresher content, has a better chance of being crawled and indexed by search engines. Always make sure your sitemap is listed at the bottom of the file, as seen in the example above. Learn more about using Jive sitemaps.
Note what changed. To upload a new version of your robots.txt file, you'll need to open a Jive case. Jive hosting teams will assist with these cases. It might be helpful to include notes regarding the changes.
Monitor your traffic. You'll need to have Google Analytics connected to your community to monitor your traffic. Upgrading to a newer version of Jive? Thinking of changing your community architecture? Always make sure your robots.txt file isn't blocking pages you want crawled. If the robots.txt file is blocking the pages you want crawled, there may be an unexpected drop in organic traffic.
Test the blocks. Google Webmaster Tools provides a free tool that allows you to check for your files and directories. If you're not sure if a page is blocked, use the robots.txt Tester to be sure.
A note about Noindex
Noindex is the preferred method to prevent pages from being displayed in organic search results. While far better than robots.txt, this meta tag cannot be applied to community pages at this time (Jive 7 or lower). As an alternative, leverage your robots.txt file.