Version 3

    Summary

    Jive Hosted and Cloud customers will are granted access to monthly reports that break down the server access logs to give additional data about the type of page load activity happening in their community. This is a built-in service that all Hosted and Cloud customers are granted. Jive is not able to generate these types of monthly reports for On-Premise instances.

     

    This document outlines the steps On-Premise customers can take to also make use of these page view reporting tools.

     

    This is not a process that Jive Support will provide any assistance on, nor will Jive be responsible for fixing/addressing any related issues for this particular document as this falls outside of our scope of support.

     

    Instructions

    1. We strongly recommend these scripts are not run on a production server.  If necessary, copy/move your jive-httpd-access.log(s) and jive-httpd-ssl-access.log(s) that is to be analyzed off your production webserver into its own empty directory. The following requirements need to be met for the scripts to execute properly.
      • Required apache log format for log parsing:  "LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D %k \"%{Referer}i\" \"%{User-Agent}i\" \"%{Content-Type}o\" %{JSESSIONID}C %F" common
        • *Note: x-forwarded-for header is needed for on-prem sites with load balancing or proxying in order to capture the original requesting IP. The script will still execute without its inclusion, however, any reports tracking IP's will be of little value without the original request IP's record in the httpd logs.
      • Required log file name:  Log file names must include  "jive-httpd-" in the file name (ie: *jive-httpd-*)
      • Required log rotation:  Daily log rotation for monthly PV reporting option. (*Note, PV results may be inaccurate if logs are not rotated accordingly. )
      • Required python interpreter: For non-jive-production systems, please create a symbolic link to python at /usr/local/jive/python/bin/python
        sudo mkdir -p /usr/local/jive/python/bin/
        sudo ln -s </path/to/python> /usr/local/jive/python/bin/python
      • Required math library:  bc - A standard math library available by command line option. (Installed by default with most linux/unix systems. Please verify, as script calls bc for mathematical precision calculation. Available through yum/apt-get and most distro repos.
    2. Copy the 4 scripts/files provided by Jive into the same folder location and unzip them. Also ensure that the system has access/permissions to the logs you will be analyzing if its not stored locally.
      • loginfo.sh - Shell script used to specify dates and type of log analysis report to produce
      • parser.py - Python script called by loginfo.sh that parses apache log files based on values passed from loginfo.sh
      • bots - Plain text file containing name's for known bots. Used by parser.py to categorize those log entries as "botviews".
      • hosts - Plain text file containing IP's of known hosts. Used by parser.py to blacklist hosts from the log analysis report.
    3. Ensure the loginfo.sh script is executable.
      chmod +x loginfo.sh
    4. Execute the loginfo.sh script on the server logs to be analyzed.
      • It can be used for a single day or for the full month.
      • It can handle multiple *jive-httpd* log files (for handling rotated logs and/or central httpd log repository for multiple webservers).  Compressed (.gz only) and uncompressed.
      • To specify your logs location, use the -l "<location>" option as shown below.  (If no path is specified, the script looks for *jive-httpd-* files in /usr/local/jive/var/logs/.)
      • The script first searches for which log files contain relevant dates to report on and then parses each log entry. Script run time performance will be effected by number of logs and log sizes being analyzed.

    Usage: loginfo.sh [OPTION]...

    Example: ./loginfo.sh -d 5/15/2014 -p apiviews -t topagents

    Options include:

      -l <location>     For on-prem Customers. Specify the path of the log files you wish to analyze. NOTE: <location> must be surrounded by "" when passed through -l

      -d <MM/DD/YYYY>   Specify date. If you do not specify a date it will assume you want the logs from yesterday.

      -m                Analyze the full month, not just the specific day.

      -p <logtype>      Specify parser script log type. Valid options include userviews, apiviews, botviews, mobilegatewayviews, mobilebrowserviews.

      -t <outputtype>   Specify the output type. Valid options include toppages, topips, topresponses, topsessions, topagents, topreferrers.

      -o <lines>        Specify the number of lines you want in the output. Default is 25.

      -u                Enable analysis on a specific url page. Copy address and put in " ".

      -f <logfile>      Enable logging and specify the file to write to.

      -w                Enable new APIview whitelisting feature. For testing.

     

    Example 1: Specify *jive-httpd* log(s) location, and report on mobilebrowserviews and show top 25 pages requested for day of  11/01/2014 and log report to "./day-report.log"

    ./loginfo.sh -l "/Share/Home/ethan.dang/temp" -d 11/01/2014 -p mobilebrowserviews -t toppages -f day-report.log

     

    Example 2: Specify jive-httpd* log(s) location, and report on userviews, and show top 50 topips for day of 11/01/2014, and log report to "./day-report.log"

    ./loginfo.sh -l "/Share/Home/ethan.dang/temp" -d 11/01/2014 -p userviews -t topips -o 50 -f day-report.log

     

    Example 3 with output sample:  Report on mobilebrowserviews and show top 25 pages requested for full month of November and log report to "./month-report.log". Use default logpath for on-premise production system, (by not specifying log location)

    ./loginfo.sh -d 11/01/2014 -m -p mobilebrowserviews -t toppages -f month-report.log

     

                root# ./loginfo.sh -d 11/01/2014 -m -p mobilebrowserviews -t toppages -f month-report.log

                logpath: /usr/local/jive/var/logs

                onprem: true

                Date requested: Nov 2014

                Searching logs for data to analyze...

                !

                Files that have log entries of interest:

     

                /usr/local/jive/var/logs/jive-httpd-ssl_Nov_sample-access.log

     

                Analyzing logs...

                python parser.py -v mobilebrowserviews None  /usr/local/jive/var/logs/jive-httpd-ssl_Nov_sample-access.log

     

                --- Official Parser Output ---

                Pageviews: 130057

                Botviews: 0

                Userviews: 23705

                APIviews: 106352

                MobileBrowserviews: 94

                  iOSviews: 90

                  Androidviews: 0

                  BlackBerryviews: 3

                  Windowsviews: 0

                MobileAppviews: 0

                  iOSviews: 0

                  Androidviews: 0

                ------------------------------

     

                Here are the 25 toppages of type mobilebrowserviews:

     

                48  /api/(51.06%)

                35  /social/rpc (37.23%)

                25  /rpc/ (26.59%)

                23  /mobile/mobile-redirect.jspa (24.46%)

                21  /api/(22.34%)

                10  /api/core/v3/activities (10.63%)

                9  /api/version (9.57%)

                9  /api/core/v3/metadata/ (9.57%)

                7  /api/core/v3/people/e (7.44%)

                7  /api/core/v3/people/ (7.44%)

                7  /api/core/v3/people/ (7.44%)

                6  /__services/v2/rest/activity-stream/new/count (6.38%)

                6  /api/core/v3/people (6.38%)

                6  /api/core/v3/metadata/properties/jive.coreapi.disable.binarydownloads.mobileonly (6.38%)

                6  /api/core/v3/members/places/471662 (6.38%)

                6  /api/core/v3/activities/recent/places (6.38%)

                6  /api/core/v3/activities/recent/content (6.38%)

                6  /api/core/v3/activities/frequent/people (6.38%)

                5  /mobile/mobile-access.jspa (5.31%)

                5  /api/core/v3/people/@me/streams (5.31%)

                4  /api/core/v3/search/contents (4.25%)

                4  /api/core/v3/places/471662/activities (4.25%)

                4  /api/core/v3/people/238170/activities (4.25%)

                4  /api/core/v3/metadata/properties (4.25%)

                4  /api/core/v3/metadata/objects (4.25%)