7 Replies Latest reply on Apr 17, 2012 9:21 AM by karl.rumelhart

    Will CMR Account for Ways People Actually Consume Jive Content?

    Ted Hopton

      We're still on 4.5.x, so my primary concern at the moment is that many people are consuming content from our Jive community via email notifications, yet these "views" are not counted in either the SBS Analytics reports or the new Community Manager Reports. Quite simply, they need to be. Without this data, we have no visibility into a significant amount of content consumption. Of course, many of us have said this before, and I saw a thread promising this will be addressed in a future release. Could we get an update on when this will happen, and if it will be part of the CMR?

       

      We will be upgrading to Jive 5 this spring, and the new UI presents another reporting challenge. Instead of "going to" a piece of content or place in the community, in Jive 5 people can expand items in their activity streams to consume (and respond) right from that screen. How is this activity being captured and reported? Will it become part of the CMR? It needs to be captured and it should be captured separately, so we can see exactly how much of this activity is taking place in streams. That will be valuable data as we move forward in developing our communities (e.g., do we need to assume that people rarely "go" to places and thereby see other content that is displayed or highlighted there?).

       

      Here's what I would like to see:

      • A new chart in the CMR that captures this kind of activity, including
        • Number of email notifications sent
        • Number of views in the community (where people navigated to the community's content)
        • Total views of community content in activity streams, broken out two ways:
          • Number of views of community content in activity streams (based on content being changed from unread to read status, so it was at least scanned or the person chose to ignore it -- we can see the ratio of this number to the number of watches to learn how many members are engaging in the activity stream)
          • Number of expanded views of community content in activity streams (each time content presented in an activity stream was clicked on to expand it)
        • Total community activity (sum of email notifications sent + views in community + total views of content in activity streams)
        • In addition to the currently offered filters, this chart needs a filter for content type, too, so we can see how documents, discussions, blogs, announcements, polls, etc., are consumed.
        • There should also be a filter to break out mobile activity versus regular browser activity, as we need to understand the ways people are interacting with the content.

       

      This chart would show the Community Manager at a glance how people are consuming content in the community. It would be invaluable in planning how to increase engagement and meet the needs of community members.

       

      Then there should be a chart similar to this, except it's for contributing activity. How are people contributing content?

      • in the community itself
      • via email
      • in activity streams
      • mobile vs regular browser filtering, and content type filtering, same as above

       

      I'd also like the User Adoption chart to add two key data points:

      • Number of people with watches enabled for the community
      • Number of people with email notifications enabled

       

      This will give a more accurate picture of how many people are interacting with the space.

       

      It's always going to be a moving target, but every change in the UI needs to be accompanied by corresponding changes in the tracking and reporting capabilities. And, of course, we need the CMR to give us more comprehensive and flexible date ranges. If there are performance concerns about displaying more than a year's worth of data on screen, then give us the option to download larger datasets without displaying them. We need to be able to work directly with this data, and it can't be limited to the past 12 months.

       

      What did I miss?

        • Re: Will CMR Account for Ways People Actually Consume Jive Content?

          I had been reminding the community owners from day 1 that email notification impacts the metrics.  We run a report to find who has a watch on the community since they are getting 100% of he content without a visit.

           

          You pointed up some key questions with the activity stream that I have asked about during the beta.  We also now have more ways to see the content (following, tracking, mentions, as well as email notification)

           

          How does all of these fit into the metrics?  Activity (creation of content) is a key item to track (more so than view). Are the metrics tracking all of these interactions and reporting them?

           

          Great question.

          Mike

          • Re: Will CMR Account for Ways People Actually Consume Jive Content?
            karl.rumelhart

            You hit the nail on the head as always, Ted.    This is a huge and important area.  I will also admit that it is hard (though that isn't your problem).  

             

            Let me explore a bit of this with you.   First regarding Views.

            * Regarding email notification, it sounds like in your opinion we can key off an email being sent (not read).  That is nice and concrete, though it isn't a dependable way of knowing that someone actually read the item.  Is that OK?  Note that many users do BOTH -- get an email and read it directly in the Web UI.  So we need to think about dups.

            * Not sure that I really follow your thoughts on how to count Activity Stream Views.  The concept of 'Read' in the stream is very loose.  Someone expanding an entry is a good signal but isn't necessary for short items so that could get kind of weird. Is this really going to be valuable or would it be just as meaningful to report on how many people are following particular activity?

            * Totally with you on Mobile.  Looking into that.

             

            Regarding how people are contributing content: makes complete sense to me but how do you prioritize info on Contribution versus more detail on Views / Follows as above?   I guess I can see breaking out contributions by email would be quite interesting.  Mobile, of course, will be massively tipped to read versus write. 

              • Re: Will CMR Account for Ways People Actually Consume Jive Content?
                Ted Hopton

                Hey, Karl, thanks for weighing in. I sure don't have all the answers on this, but I have lots of questions

                 

                Email Notification:

                • there is no way to know if someone read an email, just if there is no way to know if they skimmed it or thought deeply about it. I am happy to report that people were sent emails and confident that our authors would appreciate knowing how many of those went out. In an imperfect world, those are as good as views if you ask me.
                • the duplication is an issue, but no more so than the fact that currently we can only report views, not unique visitors. No reason to avoid the former as long as we suffer from the latter. Go to the head of the class when you solve both!

                Activity Stream views (I've edited this -- changed my mind since I first posted here):

                • I'm looking to keep things simple and approximate. Until all users wear goggles that record their actual eye movements, I'll settle for proxy actions.
                • Expanding an item in the stream is a pretty reliable indicator of "read" content. You've got a point about short items, but I just looked at my Communications activity stream and all I see are titles or content unless I expand each item.
                  • This raises the issue that different activity streams display content differently. In the Comms stream, I don't want to count content as "viewed" unless it has been expanded. In fact, if someone clicks "mark read" without expanding it, I want to disregard that, as they did not really read it.
                  • It may be that short items in the What Matters stream can be read without expanding them. I'm not sure what to do about them.
                • When I first posted here, above, I felt that if something appears in my activity stream and I change the status to "read" then it should count as a view. I no longer feel that way. "Mark as read" really means, "dispose of this before I read it." So, we should not be counting everything in the activity stream as having been read.
                • It follows from this that "followed" counts are not a valid measure of consumption, either. Stuff I follow goes into my activity stream, but I will often clear out my activity stream without reading everything in it -- who can read it all?!?! So, we'd better not tally "follows" in the same bucket as views. I think we need to monitor the activity stream, instead, and what the user actually did there.

                Contributing content:

                • are you asking me which is more important, getting more detail on consuming content (views, etc., as discussed above) or getting more detail on contribution? Given that consuming activity is more than 90% of all community activity, if I have to prioritize then I would start with more detail on consuming. But let's not overlook the power of contributors, as they make the community world go around, so the more we understand about what they are doing and how, the better we can support and encourage them.

                 

                Message was edited by: Ted Hopton

              • Re: Will CMR Account for Ways People Actually Consume Jive Content?
                nikhilnulkar

                Couldn't agree more, Ted! Thanks writing up all the details of the requirements. Internally we have discussed this too and is something we would really really love to have. Particularly stuff around email metrics as we too have a large set of users accessing via email.

                 

                Would love to hear from Jive, of what they think of these and if and when we can expect these part of CMR or otherwise.

                 

                 

                  • Re: Will CMR Account for Ways People Actually Consume Jive Content?
                    karl.rumelhart

                    Super.  Great stuff.  Let me take it a little at a time.

                     

                    Email Notification:

                    • there is no way to know if someone read an email, just if there is no way to know if they skimmed it or thought deeply about it. I am happy to report that people were sent emails and confident that our authors would appreciate knowing how many of those went out. In an imperfect world, those are as good as views if you ask me.
                    • the duplication is an issue, but no more so than the fact that currently we can only report views, not unique visitors. No reason to avoid the former as long as we suffer from the latter. Go to the head of the class when you solve both!

                     

                    This is good news (for me).  Tracking who gets sent an email is feasible.  Trying to figure who actually reads it is a can of worms I would not like to tackle.  I actually think we can deal with the dups issue.  Let me emphasize that in CMR we do report on unique visitors!  When we show a certain number of Active users, those are unique people.  It is not just a count of activities.   Similarly, when we show Content Charts these are actual content totals where we have taken into consideration things like content moving from place to place.  (This was actually very hard.)    Another small but non-trivial example of the de-dupping we do is on Questions and Answers.  We don't just tell you how many responses there are to questions or how many times the mark 'Helpful' was applied.  We tell you how many Questions have no Response, how many have at least one Response, how many have at least on Helpful response, and so forth. 

                     

                    • Expanding an item in the stream is a pretty reliable indicator of "read" content. You've got a point about short items, but I just looked at my Communications activity stream and all I see are titles or content unless I expand each item.
                      • This raises the issue that different activity streams display content differently. In the Comms stream, I don't want to count content as "viewed" unless it has been expanded. In fact, if someone clicks "mark read" without expanding it, I want to disregard that, as they did not really read it.
                      • It may be that short items in the What Matters stream can be read without expanding them. I'm not sure what to do about them.

                    There is no question that figuring out how to accurately track consumption via streams is an important challenge.   If I was to get all hip for a second (though I am not ), I might argue that the whole notion of *going* someplace to get your info is rather old-school.   The cool kids have information come to them, often via streams.   Trouble is, the very nature of this passive consumption makes it hard to track.   I hear what you are saying about expanding items in streams and so forth but not only is that technically difficult to measure (since nothing is really happening except on the screen), it also feels a bit squirrely since it depends on a particular form of a UI that isn't really intrinsic; e.g. it could be different on Mobile or another client.

                     

                    It follows from this that "followed" counts are not a valid measure of consumption, either. Stuff I follow goes into my activity stream, but I will often clear out my activity stream without reading everything in it -- who can read it all?!?! So, we'd better not tally "follows" in the same bucket as views. I think we need to monitor the activity stream, instead, and what the user actually did there.

                    I agree with you that Following something isn't really a measure of consumption.  On the other hand, it is at least a measure of potential consumption.  To me it feels similar to the email question: I don't know if you read it, but at least I know I sent it to you.

                     

                    Given that consuming activity is more than 90% of all community activity, if I have to prioritize then I would start with more detail on consuming. But let's not overlook the power of contributors, as they make the community world go around, so the more we understand about what they are doing and how, the better we can support and encourage them.

                     

                    We are on the same page.  I absolutely agree.

                  • Re: Will CMR Account for Ways People Actually Consume Jive Content?
                    karl.rumelhart

                    Ted Hopton: I have a correction and an apology to make.  I have just found out that, contrary to my previous understanding, a user selecting Preview in the activity stream DOES trigger a View.  Just the fact of an item appearing in the stream, however, does not trigger the View.