13 Replies Latest reply on Oct 23, 2015 6:51 AM by Dennis Pearce

    Measuring "working out loud"

    Dennis Pearce

      Looking for a little feedback from all you smart people here.  I've been doing a lot of thinking lately about Bryce Williams' working out loud concept, and in fact recently wrote a couple of blog posts related to it.  I think it's a very rich concept with many angles that could be explored.  I have a couple of reasons for this interest -- I'd really like to get my company moving further in that direction because I think there is tremendous competitive advantage, and also I have an idea for a doctoral dissertation related to it.  I've been trying to get a PhD part-time for years now (I'm way too old for it to have any impact on my career, but I don't want to see all that time spent go to waste ).  I kept messing around with various survey ideas related to collaboration for my dissertation proposal, but none ever really gelled for me.  I have one now that seems interesting and potentially useful, but I wanted to get some feedback before I pursue the idea any further.


      It seems to me there are two kinds of working out loud (I'll call it WOL from now on), individual and group, and they provide different kinds of benefits:


      • Individual WOL is narrating your work.  It's blogging, tweeting, posting status updates, sharing your ideas.  If enough people in an organization are doing it, it could make the organization much more innovative because there would be a rich stew of ideas floating around, ideas colliding against each other with even more new ideas forming from those collisions.

      • Group WOL is working openly as a team.  All the conversations, decisions, task assignments, etc. that previously happened in email are happening on a social platform that others in the organization can see.  Scaled up, this should make the enterprise much more efficient at getting things done since everyone is up to speed all the time, others who have relevant information but are not part of the team have a chance to provide it, and the logic behind decisions is transparent.


      So suppose we had a very short survey that could assess these two aspects of WOL (my goal would be to make it as short and simple as possible while still being valid).  Imagine two sets of questions that employees would answer using, say, a 5- or 7-point scale (these are just examples to illustrate the concept -- the real questions would probably need to be worded more carefully), something like these:



      • I blog on my internal social platform
      • I keep others informed of what I am thinking and doing using my social platform
      • It's easy for me to find out what others are thinking and doing



      • My team/dept solves problems together on our social business platform
      • Project status and communication happens on our social platform rather than email
      • We make decisions collaboratively and document them on our social platform


      With the right statistical gyrations to validate things, you can take three or four survey questions and average them into an overall score.  So if we had a score for Individual WOL and one for Group WOL, we could construct a 2x2 matrix that might look something like this:


      WOL 2x2.png


      My thinking here is that you could have a lot of ideas (high Individual WOL) but not be able to act on them, or you could be really efficient at getting things done (high Group WOL) but not have many ideas to work with.  An agile organization would be one that has a rich set of ideas to work with as raw material and also has the ability to act quickly on them.


      You could plot your organization's position on the chart and track it over time, and if you included some demographic questions in the survey you could also compare business areas, geographies, age groups, etc. within the same organization and see if there are any differences.  Since it creates numerical scores you could also correlate them with other business metrics to see if they have any impact or relationship.


      Is this something that would be useful to your organization as an analytical tool?  Do you see any problems with it or big holes in my thinking? Any feedback would be greatly appreciated.

        • Re: Measuring "working out loud"

          I really like this. Need to think a bit more about measuring - how, at what intervals, how do we asses non-responders (no ideas, no collaboration or just busy?) etc. Also, what would you do about the ideas people who have no outlets for collaboration? Or the groups that are good at working together but don't have enough to work on? I can't quite see how the analysis would clearly identify that. Also, identifying the issues is great but it's having the buy in to do something about it that is needed to.

          1 person found this helpful
            • Re: Measuring "working out loud"
              Dennis Pearce

              Good questions Belinda.  One of the reasons I was trying to make it short and simple is so that it could be used frequently.  The longer and more complicated a survey is, the more annoying it will be to employees when they have to take it often.  So the trade-off is that you can collect more information (and more finely detailed information) but you can't give the survey as often.


              Non-responders are a problem for every survey.  No way to know why they didn't respond without surveying them to find out.


              I think it would take more digging than just a survey in order to really understand why some people aren't working out loud.  I was thinking of this more as a quick way to periodically assess the organization to see if progress is being made.  If you see that you aren't where you want to be, you go try to find out why, come up with some corrective actions, apply them, and then assess again to see if they made any difference.

            • Re: Measuring "working out loud"

              Group WOL .....should make the enterprise much more efficient at getting things done since everyone is up to speed all the time,

              People still don't read it all.  The can read it all and there should be more involvement and greater opportunity for feedback in the early stages of planning but I would not assume that everyone is up to speed unless it is a dedicate group.

              1 person found this helpful
                • Re: Measuring "working out loud"
                  Dennis Pearce

                  Thanks for the feedback, Sharon!  I guess I should clarify that I was thinking of dedicated groups of people -- i.e. formal project teams or departments.  If they use a social platform to do their work then there should be two benefits that lead to increased efficiency:


                  1. Everyone on the project team should be more up to speed on where the project is than if they were using email for communication, because emails often end up being replied to and forwarded with recipients changing over time such that not everyone gets every piece of the story.  Posting to a discussion thread means everyone sees the same information in the same context.
                  2. Doing this work transparently means that others not directly involved in the project can also see what is going on and possibly contribute information that could speed up the project or avoid pitfalls.  I've experienced this several times in projects I've worked on.


                  When you said "people still don't read it all" I wasn't sure if you meant people don't read everything in the platform or everything related to their specific project.  I know no one will read everything posted in an active platform, but they should be reading everything posted to a project they are assigned to complete.  If not, that seems to be a problem that would be occurring independent of the platform or communication channel.

                • Re: Measuring "working out loud"

                  Hey Dennis...sorry I am a little late to the game on this one. I've been thinking a bit about it. The only thing I can think of is measuring activities where someone worked out loud against the more traditional alternative. Because the concept itself implies we're still doing work, likely work we would have done otherwise, so what is the differential in terms of speed, effectiveness, reach, outcome vs. how we would have otherwise done it?


                  For example, when I ran some of my internal "Week without email" experiments at work, I measured my week's activity by comparing reach as well as activity levels.


                  Examples (all sample data but similar to actual data I collected and shared with others):

                  • Week A I worked normally, received 300 emails, sent 100 emails, estimated 400 views on my 100 sent emails during that week. I made 6 posts in our ESN, achieved 500 total views on those 6 posts in one week with 25 total replies.
                  • Week B I worked out loud instead, received 200 emails, sent 5 emails, estimated 20 views on my 5 sent emails during that week. I made 50 posts in our ESN, achieved 7,500 total views on those 50 posts posts in one week with 80 total replies.
                  • Then I could calculate Return on Effort (ROE) in some way. There probably is a better way to do it, but most simply you could say ROE = Views / Communication. But realistically you could account for the time impact of reading incoming comms too and the impact that has on your total effort expended. Need to chew on that one and how you'd make it work best. But going the most simple route for conversation sake:
                    • Week A ROE = 900 views / 106 communications = 8.5 ROE
                    • Week B ROE = 7,520 views / 55 communications = 136.7 ROE
                  • I'm also mostly comparing against email here. WOL also has an element of putting deliverables in an observable place even if not proactively notifying people to interact with it. So accounting for unplanned discovery, reuse, input as a result of doing that vs. putting something in private or on a hard drive?


                  In other words, what's the impact of me shifting my work from what I've always done? How can I isolate the value of that shift? And I think the best way is by comparing against a previous baseline that you can measure the before and after of our personal behavior choices and changes.


                  Some factors I'd need to think about incorporating into a real ROE type formula:

                  • Weighting "Open" shared comms vs. "Private" shared comms due to great future value created as a result
                  • Effort impact of consuming incoming communications, and possibly the effort impact of "incoming" comms that are targeted to you ONLY (email or a direct mention) vs. "incoming" comms that are targeted to a community where the obligation to you is lower to actually respond as desired, not as required...particularly in a timely manner.
                  • Weighting potential outcomes in the formula. Correct answer provided, maybe with a time element (faster = better)? Number of inputs received? Number of subscribers that were proactively notified of the interaction? etc.


                  Probably more complicated than even I thought I'd go when I started typing this....but it's where I probably would start heading...more to chew on here...

                    • Re: Measuring "working out loud"

                      Bryce, I like your idea.  One thought/question I have is can you measure WOL-ROE in a linear model beyond the individual?  I am thinking for one person, perhaps, but when you expand the WOL- ROE across groups, departments, organizations, could the combined WOL-ROE be measured in looking exponentially in delivering the true value of that Shift?  Meaning the more people WOL, then the greater the ROE in the aggregate versus simply adding each individual's WOL-ROE? 


                      Just thinking out loud (TOL)


                        • Re: Measuring "working out loud"

                          Yes, when considering network effects...and the impact of "value this week" vs. creating reusable value for the next "2-3 years" by having had the interaction in a findable place. It's one thing for it to happen in the open, it's another thing for it to remain in the open and have powerful tools for finding, recommending, serendipitous discovery, resharing, etc.


                          So...yes .

                        • Re: Measuring "working out loud"
                          Dennis Pearce

                          One advantage of even private shared comms is that if they are on the platform, they can easily be made open later by changing space permissions, moving the discussion, etc.  Compare this to a similar effort in making emails open.  So even if there is no immediate benefit from sharing openly, there is still some potential benefit because of the structure of the communications.

                        • Re: Measuring "working out loud"
                          Dennis Pearce

                          I stumbled back on this today so I thought I should update it since it's been two years since I first posted.  I did end up doing my doctoral thesis on Measuring Working Out Loud and successfully defended it in December of last year. See "Developing a Method for Measuring "Working Out Loud"" by Dennis E. Pearce


                          As I mentioned above, my goal was to develop a survey with the minimum number of questions that would still be statistically valid for creating a WOL score or index.  I ended up with six questions, three each for individual working out loud and group working out loud.  All responses are in the form of a 1 to 7 scale from "Never to "Always" with the exception of the last question, which is 1 -7 from "to no extent" to "to a great extent."



                          • When I work on a team, we share the team’s goals in a way that those in other parts of the organization can see.
                          • When I work on a team, we communicate with each other in a way that those in other parts of the organization can see.
                          • When I work on a team, we make our work visible to the larger organization before it is complete. 




                          • I share my thoughts and ideas on <enterprise social platform> with others beyond my immediate co-workers.
                          • I share difficult work-related problems on <enterprise social platform> with others beyond my immediate co-workers.
                          • I participate on <enterprise social platform> by doing things such as starting discussions, making comments, creating status updates and blog posts.


                          If I did all of my statistical mumbo jumbo correctly, you should be able to take each set of three questions and average their scores together to get an Individual WOL index and a Group WOL index. I was hoping to get a single WOL index but the statistics wouldn't hold up.  In retrospect I think this makes sense because doing your work in an open environment and sharing your work really are two different behaviors and mindsets, even though they are both working out loud.


                          How might you use this survey in an organization?

                          • If you are interested in developing WOL for its own sake (see Working Out Loud: For a better career and life by John Stepper), you might survey periodically and track it over time to see how it is changing.
                          • If you want to deploy an initiative intended to improve WOL, you might do a quick survey prior to and then again after the initiative to see whether it had an effect and how much.
                          • Add some demographic questions so that you can compare countries, divisions, or business areas to each other.
                          • Run correlations against other business metrics.  For example, in my company we do an employee engagement survey every year.  I'm looking into how we can correlate that data with WOL to see if there is any relationship between WOL and engagement.
                          • WOL to share your WOL data.    If any other companies decide to use the survey, I would be interested in comparing notes to see if there are any common patterns across companies and industries.  Examples might be things like is WOL more useful for R&D than for Sales & Marketing?  Because of culture, are some countries more amenable to WOL than others?  Does WOL work better for some industries than others, or is it universally beneficial?  Those kinds of questions ... 
                          1 person found this helpful
                          • Re: Measuring "working out loud"
                            Dennis Pearce

                            FYI, I have an article on WOL that was posted to the KMWorld web site yesterday: Does Your Company Work Out Loud? - KMWorld Magazine   I included a link to Working Out Loud Turns Knowledge Into a Utility (just trying to drive a little traffic back to the JC  ).


                            I'll be speaking at their conference in a couple of weeks so anyone who might be going, let me know if you want to meet up!