24 Replies Latest reply on Dec 7, 2012 10:08 AM by tzeng

    Managing page views

    sara.perry

      We have had a frustrating year trying to manage page views to budget...

      with limited reporting and functionality available in Jive to assist us.

       

      Welcome and appreciate any tips other users have.

      And sharing my wish list:

       

      --Regular reporting that includes page views, bot views by bot, and a running total of our page views to date

      --Best practices guide for community design that helps maximize page view budget

      --Jive functionality that enables management of bot views.
      Google webmaster tools enable throttling their bots, but are inexact.
      And, so far have found nothing similar for other bots.  Baidu and Sougu, for example, seem to be an all-or-nothing proposition.

      So, my options seem to be to completely turn off crawls (so any search benefit) in Asian markets--or to allow crawls--and have no ability to manage to page view budget.

       

      Neither are attractive options...

      And this issue is key reason we would explore alternate options for community platform.

       

      I hope the community has some tips to help us address, and that Jive product team recognizes and responds to the issue...

       

      Sara

        • Re: Managing page views
          crossman

          Sounds like you would benefit from using WebTrends or Google Analytics that can install a tracking javascript code in your Jive instance - and get many of the kind of reports you are asking for.

           

          I wonder if you could install a robots.txt file on your Jive instance to manage frequency of bot spidering (assuming the bots in question respect the robots.txt file).

           

          Have you brought up these concerns with your Jive account manager?

            • Re: Managing page views
              sara.perry

              HI Kevin,

               

              We have Google Analytics on the site. Please tell me more about tracking javascript code…and how this might help.

               

              We tried robots.txt. Baidu and Sougu ignored…and racked up tons of page views.

              Reports to check which bots are crawling must be individually requested, and Jive is sometimes quite slow in responding.

              After learned the robots.txt was ineffective in managing these views we had Jive completely block these crawls…

              but this will not be a good long term solution as we expand into Asia.

              (But…not being able to manage to a page view budget is not viable either.)

               

              Yes, our account rep knows the issue well…

              and has been supportive in addressing—but is limited by the limitations of the platform.

              Your support team has also spent significant time in addressing.

               

              True fix must come from product.

               

              Sara

                • Re: Managing page views
                  erskine

                  Greetings Sara,

                  I'm the product manager in charge of this aspect of our product offering. As you point out, at present it is not possible to block bots in a convenient manner. However, our hosting team could work with you to apply rules to your apache config which can block specific bots. You can file a support ticket to get the team working on this for you. While it would be a convenient feature to provide this capability in an automated fashion, as it turns out, we have had relatively few requests for blocking at this level of granularity (maybe 5 or 6 customers), thus we haven't slotted this capability in our immediate roadmap.

                   

                  On the reporting front, we are working on new infrastructure to harvest all of the web access information and provide tools to report out on it. I don't have a concrete date for you on this, but it will be in the Q1/Q2 2012 timeframe.

                   

                  Cheers,

                  Erskine

                    • Re: Managing page views
                      sara.perry

                      Erskine,

                       

                      Appreciate the response...however I am not sure you are clear on the issue.
                      I have worked with your support team...and currently have all but Google bots blocked.

                      We did this, in part, because Baidu (and other) crawls were eating up our page views...driving us toward a new tier (and a significant bump in price).

                      As we move into Asia we will want to be found in search in Asia, but we are concerned about having absolutely no control over page views.
                      For Baidu, Sogou, and others it seems to be an all or nothing proposition.  Unlimited crawls or completely block them.

                      This makes it very hard to predict costs or manage to a budget.  I encourage you to take a look at the bot-crawl reports your team generated on our account this year...so you can see what a huge proportion of our page views were consumed by bots, and how quickly the Baidu crawls grew.  This will likely give you a clearer understanding of the issue.  It is a significant problem.  I cannot imagine that we are the only businesses seeing this.  Sara

                        • Re: Managing page views
                          erskine

                          I feel as though I understand the situation, but I'm not sure I know what the practical solution would be. I suppose we could implement configurable throttling and filtering mechanisms at our apache webserver level, but that would constitute a considerable investment given the relative paucity of requests we've had for such a feature. Most customers are content with the tradeoff between the cost of higher page views and the value derived from the resultant increase in human traffic. Are there solutions out there in the market you can point me to that offer such a capability?

                            • Re: Managing page views
                              erskine

                              I would also add that some search engines will blacklist your site if they detect any tampering with the access or offloading of requests.

                              • Re: Managing page views
                                sara.perry

                                I wish I had some solutions to point you to...

                                The how-to technology is not my area of expertise.

                                 

                                My view is from a customer perspective...

                                My Jive pricing structure is based on page views.

                                So I look for ways to predict/budget/manage the views.

                                I imagine other customers would have the same need to be able to predict costs, so ask you how to address.

                                Product help to manage bot crawls? A pricing structure that makes budgeting more predictable?

                                Is the pricing fixed if we host our own servers vs. using your SaaS solution (thus taking the real cost of page views off Jive servers)?

                                 

                                I do understand the value of bot crawls and search...

                                Have heard (from various interactions with Jive) that the number of bot crawls seems on the high side.

                                But, no insight as to why or how we might manage to get the best value from our service.

                                 

                                Welcome and appreciate any thought.

                        • Re: Managing page views
                          sara.perry

                          Hi Kevin,

                           

                          Still interested in learning more about tracking javascript code…and how this might help.

                          Please share your thoughts here.

                           

                          Thank you,  Sara

                            • Re: Managing page views
                              gordon_sorensen

                              Sara - I realize this is an old thread, but I am trying to get some info on this topic. We are an on prem customer; sounds like you might be hosted? But you ask Kevin about the jscript. I didn't see his reply, but if you go to google.com/analytics and hunt around, it is pretty easy to find the javascript code for Google Analytics. Grab it (you have to create an account first to get a number, which is easy) then you just drop it in one of the Jive system properties (named something like Third Party tracking). Our issue arises from the view counts our members see vs what we get on Google Analytics. The thing about the GA code and reporting is that it doesn't count bots and crawlers. However, the member facing side of Jive that shows view counts DOES count that. As a result, page views are grossly inflated. I have a sense everyone deals with this, but am surprised not more is done about it by Jive. BTW - we are running 4.0, but from related threads, doesn't seem like anything has been improved in 5.0 which we are considering moving to.

                                • Re: Managing page views
                                  rjbush

                                  I am interested in how people handle this as well. We are hosted on 5.0 and are getting tons of bot views.

                                    • Re: Managing page views
                                      gordon_sorensen

                                      I'm not sure there IS a way to manage it. For hosted customers, it sounds like Jive filters the bots out based on some info in related threads. From an on prem customer like us, we rely on Goog Analytics for a 'reliable' count. Unfortunately, I wish we at least had the flexibility to do something in the user interface where the enormous amount of views listed isn't close to accurate. On the one hand, I guess it might suggest to someon that we have a lot of traffic, which, een tho it is from bots, is technically true. On the other hand, those numbers are greatly inflated.

                                        • Re: Managing page views
                                          sara.perry

                                          We are a hosted customer.  It is a source of frustration that Jive charges are based on page views...but has limited tools for managing unwanted bot crawls.  Process for me is to file a support ticket request a report of all of the bot crawls...so I can see exactly which bots are crawling.  Then file another ticket to request Jive to block specific bots that crawl but provide no value to us.  Then I file another ticket about a week after they have confirmed they have blocked these bots...asking for a report for the period following the block...so I can see if, in fact, those bots are being blocked. And, if not (as they sometimes are not)...I start the process again.  It is a good idea to do "check ups" as well...as bots that were previously blocked may come unblocked for some reason.  Overall aim is to ensure we get good value from all of the page views that we are paying for...

                                            • Re: Managing page views
                                              tzeng

                                              We are in the exact same scenario - where botviews are jacking our pageviews far above what we expected and we are being pushed onto the next tier of licensing because of it.

                                               

                                              Why did you have to file three tickets at once instead of having them actually do the 3 steps?

                                                • Re: Managing page views
                                                  sara.perry

                                                  It was a series of activities.  I wanted to check what bots were crawling...and so I could selectively block.  (I did not want to block Google bots that help with search rankings, for example.)  Then I needed to request the block.  Then...I wanted to audit to ensure that the unwanted bots were not longer racking up page view. 

                                                  • Re: Managing page views
                                                    gordon_sorensen

                                                    Another thing you might want to consider is using Google Analytics to help confirm some counts, or at least eliminate the botviews. According to GA, since the GA tracking code is javascript based, it doesn’t record a bot hits a page. Don’t ask me anything beyond that as I don’t have add’l detail handy. But I also figure if bot views jack up (and in our case REALLY jack up) view counts so high, and many of us have noticed this, Jive should have some mechanism in place to factor our bot driven ‘views.’ In our case, it has been suggested it is silly to show view counts in our public comms – tho I can’t figure out how to not display them. Our private communities, being blocked from bots, don’t have that problem. Tho we only have a few of those.

                                                      • Re: Managing page views
                                                        sara.perry

                                                        Jive does not recognize Google counts...bills page views only on their own tracking.  And...Google cannot stop other boths (Baidu, for example) from crawling your site.   So, I found limited help available there.

                                                          • Re: Managing page views
                                                            gordon_sorensen

                                                            I understand that Jive wouldn’t pay any attention to Goog counts. I was just suggesting as an optional way to begin getting an idea of what real counts are. Where did you get the info that Goog doesn’t stop all bots? Especially interested in the Baidu, as I have noticed our counts for Chinese language browsers have increased a noticeable amount since January. Not sure if bots they would be counte in that manner tho …..

                                                              • Re: Managing page views
                                                                sara.perry

                                                                Google can only control their own bots. To stop Baidu crawls we had to ask Jive to put a block on our site.  All bot crawls contribute to your overall page view count with Jive.  Jive can provide you with a report to let you know exactly which bots are crawling your site and pageviews per bot. You can request this type of report with a support ticket.

                                                                  • Re: Managing page views
                                                                    gordon_sorensen

                                                                    We are an on prem customer, and on an older version, so not sure what Jive can/can’t tell us about page views on our site (and as alluded to earlier, we don’t have the Jive report module). Do you know how to detect a Baidu crawl? In the log files, but identified with biadu somewhere in the record? And maybe dumb question, but if it can’t crawl your site, it can’t return it in the results, right?

                                                                      • Re: Managing page views
                                                                        sara.perry

                                                                        Don't know about your instance.  This was not a report I was able to pull myself. Had to specially request it.
                                                                        If Jive is billing you for page views...they should be able to tell you where those page views are coming from.  We only understood the Baidu crawls from this report.  (This came after seeing pageviews skyrocket...not at all in light with Google numbers...and asking Jive for assistance in understanding where the numbers were coming from.)  Not sure I am clear on your question...but if I understand correctly:  If bots can't crawl, you won't show up in search results.  This is true as I understand it.  So...if you stop the Google bots you lose your Google results.  If you do business in China you may not want to stop Baidu crawls...

                                                                • Re: Managing page views
                                                                  tzeng

                                                                  We've been using GA since we launched, but we've definitely learned that Sara's experience is typical of this scenario - Jive Hosting only takes into account their own page view measurements for billing purposes.

                                                                   

                                                                  Our entire community is external and we've actually had concerns about page views for some documents, so we've hidden it in the theme for certain types of documents. Just some CSS dropped into the theme, not a system setting on Jive SBS itself.

                                                                    • Re: Managing page views
                                                                      sara.perry

                                                                      Jonathan,

                                                                       

                                                                      Can you share more detail about what you've done?  Sounds like it might be helpful to us.

                                                                      Thank you,  Sara

                                                                        • Re: Managing page views
                                                                          tzeng

                                                                          It was a purely cosmetic fix - it doesn't change anything about page views tracking/recording.

                                                                           

                                                                          We have documents that we market to a wide audience and when these are first posted on the site, obviously there are few low view counts. Since low page views might be perceived in a negative light by viewers, we simply avoided the issue by removed the page view counter altogether.

                                                                           

                                                                          In slightly more technical description, we have an id, lets call it XXX, in our theme that applies itself when documents are created via a custom workflow. Our theme then reads as "any documents tagged as XXX ID should appear without the page view counter." There are other elements that we can then attach, but it's one way to separate a group of documents so that we can group them and have them display a specific way without affecting documents we want to display in the default Jive manner.