2 Replies Latest reply on Mar 13, 2015 2:27 AM by varunluthra

Help with robots.txt configuration

varunluthra Mar 12, 2015 11:58 PM

We need google to stop crawling this page: /people/varunluthra/people but crawl this one https://community.jivesoftware.com/people/varunluthra

We need to update robots.txt file, if we add this /*people$ in our robots file, would it stop crawling both or just the first one highlighted in red?

Re: Help with robots.txt configuration

mcollinge Mar 13, 2015 2:08 AM (in response to varunluthra)

You can test different patterns in Google Webmaster Tools.. simply edit the robots.txt file in the browser, and test whether the URLs are blocked or not. The changes won't be actually applied to your live robots.txt file.
Like (0)

Actions
- Re: Help with robots.txt configuration
  
  varunluthra Mar 13, 2015 2:27 AM (in response to mcollinge)
  
  Can't I just check it without having the webmaster access for this site?
  
  Like (0)
  
  Actions

Go to original post