Version 1

    Going Boldly..

    As with all things sometimes a need exists to do something that is not available. In this case, you may have a need or be tasked with creating bespoke reports for a Jive Community. We illustrate here the resources that are available and how to leverage these to create reports of interest. What this document does not do is describe the configuration or authentication processes for any of the described services although where appropriate recommended excellent resources are referenced. Take time to complete these references they have invaluable information.

     

    For the example we'll use Node JS here for simplicity, refer to Getting Started > Installing the Jive Node SDK

     

    What's available

    Several years ago Jive developed the Data Export Service (DES), this is a highly performant cloud based service that will allow you to rapidly extract all activity happening in your Jive Community. It allows you to specify time frames and other important filters such as activity type and place. It is a hugely powerful tool for report creation. Refer to Using the V2 Jive Data Export Service (Analytics) on getting started.

     

    The Jive REST API is a mature highly leverageable API exposing Jive's functionality, data, content and configuration as a report developer (Adventurer) it is a useful tool when used in conjunction with DES. The Jive REST API is fully documented Jive REST API v3.14.

     

    At the beginning of this article we mentioned Node JS. It is a great tool it allows you the Adventurer to consume and collate data using public libraries for your own purposes. See npm there are many useful libraries to consider. During report creation collation and filtering can be facilitated by using  for example the Underscore.js library. We will review a few examples further along in this document.

     

    Start the Report

    For the purposes of this Adventurer's Guide we'll strive to consider how to would build a 'Place Activity' report, essentially providing a hit list count for places in a Jive Community.

     

    e.g.

     

    containerId,containerType,containerName,groupType,creationDate,modificationDate,creator,creatorEnabled,activityType,activityCount,users
    4846,socialgroup,Health - First Challenge,PRIVATE,9/5/2016,9/5/2016,AshleyW,true,ALL,4,1
    4844,socialgroup,Academy Part 1 - September 2016 PH & VN,PRIVATE,9/5/2016,9/5/2016,DavidN,true,ALL,15,1
    4840,socialgroup,new cloud group,SECRET,9/3/2016,9/3/2016,StuartB,true,ALL,7,1
    

     

    Grok with JSON

    For this example, Yes! we'll be working with JSON it gives a great object model that can be rapidly consumed and is lightweight and readable. In addition compared to the CSV export there is more data available in the JSON object(s) returned from DES or at the very least it is certainly seems formatted in a more readable manner.

     

    Get All Rows

    It has been the current experience that it has not been necessary to restrict the number results since we specify a before and after date time (ISO yyyy-mm-dd) typically, as a rule it is good practise to specify all rows.

     

    {ANALYTICS_PATH}?count=all
    

     

    Optimise the Query

    Assuming that the report that is being delivered content is entirely available in DES the first element to address is to minimise the data returned leveraging DES's extraction abilities. Important factors are the time field, the content type, and the scope of the query within Jive if applicable (place). Note DES requires the ISO Date Time Format

     

    {ANALYTICS_PATH}&after=" + startDate + "&before=" + endDate
    

     

    Specify what you're after

    For the purpose of our example report restrict our report to 'views' only. This we achieve by setting the activity name to filter upon, this means we ignore likes, comments, logins, logouts and a plethora of activities irrelevant to use for this report. Here is also another important element note that the ACTIVITY_VIEW_DOCUMENT encompasses several content types not just documents.

     

    {ANALYTICS_PATH}&filter=name(ACTIVITY_VIEW_DOCUMENT)

     

    Complete Example URL

    Putting it all together the URL that has been built could look something like the below.

     

    https://api-eu.jivesoftware.com/analytics/v2/export/activity?count=all&after=2016-04-01&before=2016-05-01&filter=name(ACTIVITY_VIEW_DOCUMENT)

     

    Got the data, now what?

    I appreciate not everyone is a Node JS or JSON parsing guru. Node JS makes available a global JSON object for you to parse all the data that has been received. From which it is a simple task to reference the various fields of the parsed object.

     

    var data = JSON.parse(body);
    if (data) {
       var paging = data.paging;
        if (data.paging) {
            logger.info("** Total Indvidual Result Count ** : " + data.paging.totalCount);
        }
    }
    

     

    Take note of JSON.parse on line 1 (it rocks).

     

    It is useful to have the plain JSON output (copy, paste or use browser) to hand when analysing the data returned and the fields that are relevant. That's a huge amount of work done for free. Essentially one now has a memory model of the data returned now to manipulate at will. Again the references above provide good examples of the JSON payloads to be expected.

     

    Building a report

     

    Get a relevant object model

    In the writer's experience it is best practise to form an object as closely formed to the representative report model as possible. To this end we will count each place as having an activity count, if we already have this in a map we'll increment the hit count (count++)

     

    function populatePlaceResults(list) {
        for (i = 0; i < list.length; i++) {
            var activity = list[i].activity;
            var actionObject = activity.actionObject;
            var actorID = list[i].actorID;
            var destination = activity.destination;
            if (destination){
                var placeId = destination.objectId;
                var creationDate = toDateTime(destination.creationDate);
                var modificationDate = toDateTime(destination.modificationDate);
                if (!(placeId in result) && (destination.objectType != "usercontainer")) {
                    var place = {
                        containerId: placeId,
                        containerType: destination.objectType,
                        containerName: destination.name,
                        groupType: undefined,
                        creationDate: creationDate,
                        modificationDate: modificationDate,
                        creator: undefined,
                        creatorEnabled: undefined,
                        activityType: (specifiedActivity == undefined)? "ALL": specifiedActivity,
                        activityCount: 1,
                        users: [actorID]
                    };
                    result[placeId] = place;
                    asyncTasks.push(readPlaces.bind(null, place.containerId, getObjectTypeId(place.containerType)));
                } else if (placeId in result){
                    result[placeId].activityCount++;
                    var users = result[placeId].users;
                    if (users.indexOf(actorID) == -1) {
                        result[placeId].users.push(actorID);
                    }
                }
            }
        }
    }
    

     

    The Push & Bind

    An astute reader would have noticed an array populated with readPlaces with a matching bind call (line 26). The bind call, binds parameters to a function to be called at a later time (note these are pushed by value). This is being done to get some additional data for the report via the Jive REST API. This data is added to enrich out report object result. Reference async/README.md at v1.5.2 · caolan/async · GitHub. Note the need to call the callback method, this notifies async library that the asynchronous operation has completed successfully. Failure to do so will cause the node script prematurely. Also beware of callback pyramid hell. There are lots of good references out there but for the readers reference please see Avoiding Callback Hell in Node.js.

     

    function readPlaces(placeId, typeCode, callback) {
        var url = PLACE_URL + "?filter=entityDescriptor(" + typeCode + "," + placeId + ")";
        request.get(url, function(error, response, body) {
            if (!error && response.statusCode == 200) {
                var groupList = JSON.parse(body);
                if (placeId in result) {
                    groupList.list.forEach(function(group) {
                        try {
                            if (group.hasOwnProperty("creator")){
                                logger.info("Found creator info for place: " + group.displayName);
                                result[placeId].creatorEnabled = group.creator.jive.enabled;
                                result[placeId].creator = group.creator.jive.username;
                            }
                            if (group.hasOwnProperty("groupType")){
                                result[placeId].groupType = group.groupType;
                            }
                        } catch (err) {
                            logger.error(err);
                        }
                    });
                } else {
                    logger.error("Crazy error the place Id is not in the result array: " + placeId);
                }
                callback(null);
            } else if (!error && response.statusCode == 404) {
                logger.debug("security?");
                callback(null);
            } else {
                logger.error("Couldnt resolve " + placeId);
                callback(null);
            }
        }).auth(ADMIN_USERNAME, PASSWORD, true);
    }
    

     

    Do things in Synchronously

    The aforementioned async methodology is an important element to the solution, to not overrun the maximum number of threads in the Jive instance's web container. It can be seen in our example the report is executing the JIVE REST API requests in series. This is a Jive resource friendly way to not intrude while executing the report,

     

    function doAsyncTasks() {
        async.series(asyncTasks, function() {
            logger.info("Tasks complete ok.");
            handleResult(result, "place_activity.csv");
        });
    }
    

     

    A rapid way to report

    Since the result object model is the data required for the report to create the required CSV output file for this report is merely a case of outputting the data using a csv file library.

     

    function handleResult(result, filename) {
        result.forEach(function (obj){
            obj.users = obj.users.length;
        });
        sorted = _.sortBy(result, 'numberOfViews');
        sorted = sorted.reverse();
    
        logger.info("Creating csv output file for user search results");
        var csvStream = csv.createWriteStream({headers: true}), writableStream = fs.createWriteStream(filename);
    
        writableStream.on("finish", function(){
          logger.info("Finised writing results to file");
        });
    
        csvStream.pipe(writableStream);
        sorted.forEach(function(object) {
            if (object != undefined){
                csvStream.write(object);
            }
        });
        csvStream.end();
    }
    

     

    As can be seen above the underscore library (_) on line 5 is used to sort the result data, as powerful mechanism, see the call to reverse. Note is also possible to manipulate the headers etc. Another neat trick is the transform method which will allow you the Adventurer to mould your report data further if needed e.g.

     

    csvStream.transform(function(obj) {
        return {
            SearchString: obj.searchFinalQuery,
            TimesSearched: obj.hits,
            Users: obj.actorIDs.length,
            actorIDs: obj.actorIDs
         }
    });
    

     

    Other Things..

    Using the Underscore library  it is possible to manipulate your report data prior to output, for example below.

     

    Collating, Grouping

    Using the underscore groupBy function for JSON objects in a array a rapid way to group is facilitated. In the example it is demonstrated that this can be nested. It may be advisable for very large data sets to perform this asynchronously.

     

    function handleResult(place, result) {
        console.log("Handling result, we group and collate here...");
        var final = [];
        var grpdByAuthor = _.groupBy(result, 'author');
        for (var author in grpdByAuthor){
            var grpdAuthorObject = _.groupBy(grpdByAuthor[author], 'objectId');
            for (var object in grpdAuthorObject){
                grpByAuthorObjectActivity = _.groupBy(grpdAuthorObject[object], 'activityType');
                for (var activity in grpByAuthorObjectActivity){
                  collatedObject = grpByAuthorObjectActivity[activity][0];
                  collatedObject.activityCount = grpByAuthorObjectActivity[activity].length;
                  final.push(collatedObject);
                }
            }
        }
        // sort
        sorted = _.sortBy(final, 'activityCount');
        sorted = sorted.reverse();
        // report
        writeToFile(sorted, "who_did_what_where.csv");
    }
    

     

    Filtering, Finding and Updating

    Specific results of interest also easily extractable using underscore again as below.

     

    function updateUserMemberships(){
        logger.info("Starting updating memberships");
        for (var personId in result){
            var securityGroups = _.where(securityGroupMembers, {memberId: personId});
            securityGroups.forEach(function (group) {
                person.userPermissionGroups.push(group.name);
            });
        }
        logger.info("Finished updating memberships");
    }