Skip navigation

This post is the third in a series of blog posts about customizing for Clearspace 2.x. The previous posts covered general information about Customizations in Clearspace 2.x and upgrading themes and FTL files. This post continues the series with more information about widgets.

 

Widgets can be used much more broadly in version 2 than previously, so most of your widget-specific upgrade changes are related to this broader support. In version 1 a system or space administrator could use widgets only to assemble a space or community overview page. In version 2 both administrators and users can customize page layouts in several areas. Widgets can be used on the Clearspace instance home page (which has been rendered from main.ftl), a user's personal home page, a community/space overview page, a project overview page.

 

Note that unless your widget is extremely simple, you're likely to also need to keep in mind API changes and FreeMarker changes. For example, version 2 requires that widgets acquire references to Clearspace manager beans using Spring. The rest of widget development model  -- artifacts involved, how you deploy them, and so on  -- is unchanged from version 1.

 

When upgrading a widget (or developing a new one on version 2), you use the @WidgetTypeMarker annotation to specify which of the display contexts your widget is allowed in. Deciding which contexts to allow is an important part of your widget's design. For example, you might decide that a widget that takes a very individual-oriented set of property values (such as the Tag widget, which displays content associated with a particular tag) isn't useful in high-level contexts such as the instance or space home pages (where the broad set of people viewing might want views on a large variety of tags).

 

The @WidgetTypeMarker annotation supports values from the WidgetType enumeration. Here's an example that includes all of those values:

 

@WidgetTypeMarker({WidgetType.HOMEPAGE, WidgetType.PERSONALIZEDHOMEPAGE, 
    WidgetType.COMMUNITY, WidgetType.PROJECT})
@PropertyNames("myProperty")
public class MyWidget extends BaseWidget {
    // Implementation code omitted.
}

In order to reduce the tight coupling between the widget views (which are typically built using FreeMarker) and the rest of the Clearspace application, the widget context that was previously populated with a reference to the current context via a JiveContext instance has been modified so that the widget view no longer has access to that instance. That means that you'll need to provide everything that your widget view needs through the properties that the widget interface requires. Typically you'll create a widget class that extends BaseWidget and then you'll override this method:

 

protected Map<String, Object> loadProperties(WidgetContext widgetContext, ContainerSize size)

So in the previous version of Clearspace, you might have had something like this in the FTL file that's your widget's view:

 

${widgetContext.jiveContext.communityManager.rootCommunity.ID?c}

Because the JiveContext instance has been removed from the WidgetContext class, you'll need to provide your view with the properties it needs explicitly in your widget. Here's an example:

 

public class MyWidget extends BaseWidget {

    // Declare a property variable and setter for injection by Spring.
    private CommunityManager communityManager;
    public void setCommunityManager(CommunityManager cm) {
        this.communityManager = cm;
    }

    protected Map<String, Object> loadProperties(WidgetContext widgetContext, 
        ContainerSize size)
    {
        Map<String, Object> properties = super.loadProperties(widgetContext, size);
        // Implementation code omitted.
        properties.put("rootCommunityID", communityManager.getRootCommunity().getID());
        return properties;
    }
}

Finally, because you can create a widget that exists in multiple parts of the application (the homepage, a personalized homepage, a community, a project), you'll sometimes want to change the behavior of your widget based on where the widget is being rendered. You can determine the render context of your widget by checking the type of the WidgetContext class that you're given in the loadProperties method. Here's some example code that shows how you can determine what context the widget is in:

 

public class MyWidget extends BaseWidget {
    protected Map<String, Object> loadProperties(WidgetContext widgetContext, ContainerSize size) {
        Map<String, Object> properties = super.loadProperties(widgetContext, size);
        if (widgetContext.getWidgetType() == WidgetType.COMMUNITY) {
            CommunityWidgetContext cwc = (CommunityWidgetContext)widgetContext;
            // Do something specific for the community
        }
        else if (widgetContext.getWidgetType() == WidgetType.HOMEPAGE) {
            HomepageWidgetContext hwc = (HomepageWidgetContext)widgetContext;
            // Do something specific for the homepage
        } 
        else if (widgetContext.getWidgetType() == WidgetType.PERSONALIZEDHOMEPAGE) {
            PersonalizedHomepageWidgetContext phwc = 
                (PersonalizedHomepageWidgetContext)widgetContext;
            // Do something specific for the personalized homepage    
        }
        else if (widgetContext.getWidgetType() == WidgetType.PROJECT) {
            ProjectWidgetContext wwc = (ProjectWidgetContext)widgetContext;
            // Do something specific for the project
        }
    
        properties.put("rootCommunityID", communityManager.getRootCommunity().getID());
        return properties;
    }
}

 

The information above along with more details can be found in the Upgrading Extensions to 2.0 documentation.

 

You can also watch a video and download a presentation to learn more about how to write new widgets for Clearspace 2.0 from Aaron Johnson, Engineering Manager at Jive.

 

 

Did you know that you can add any Clearspace instance to the list of search engines available for quick searches from your Firefox or IE search box?

 

Since Clearspace is Opensearch enabled, Firefox will automatically recognize it and give you the option to add it to your drop down list of search engines.  This is made possible because we add an Opensearch descriptor to every page in Clearspace, which looks something like this:

 

<link rel="search"
    href="/opensearch.xml"
    title="Jivespace"
    type="application/opensearchdescription+xml"/>

 

This works just as well for internal instances of Clearspaces (assuming you are logged into your corporate network).

 

It is as easy as this:

  • Navigate to any page of any instance that has opensearch enabled

  • Add that search engine from Firefox's drop down

 

Opensearch is already enabled by default, so you should be able to add any Clearspace installation to your browser search box. More information about using OpenSearch in Clearspace can be found in the Clearspace System Administrator's Guide.

 

 

 

This post is the second in a series of blog posts about customizing for Clearspace 2.x. The first post covered general information about Customizations in Clearspace 2.x. This post continues the series with more information about upgrading themes and FreeMarker template (FTL) files.

 

Each new version of Clearspace includes changes to FTL files and version 2 is no exception. Typically these changes are needed to support new or enhanced features. The FTL changelog included with a Clearspace release provides a list of the FTL files that were changed from the previous version. However, changes in version 2 have pretty much touched every template. In most cases these changes require a simple search-and-replace to fix. But some changes are more substantial.

 

The information on upgrading themes suggests an incremental process for making your changes that could save you some aggravation.

 

The following list suggests the tips and best practices for upgrading your templates.

 

  • When you've overridden existing Clearspace templates (as in a theme), start with new version 2 FTL files that correspond to the version 1 files in your theme. Then, working from your customized version of the version 1 template, transfer your changes to to the version 2 template. Updating the new templates might help you avoid accidentally transferring code from version 1 that no longer works on version 2. This is especially important given that FreeMarker errors are difficult to debug; errors don't show up until run time. Weaving your version 1 changes into the version 2 templates will make the process more systematic.As you copy your changes into the version 2 template, keep in mind the changed items described below.

  • The version 1 pages whose templates were community.ftl and main.ftl -- community pages and the home page -- can be easily customized with widgets in version 2. Before you set out to upgrade these pages, take a look at how much of your customization work could be accomplished by using widgets on the version 2 templates. Using widgets might reduce the amount of customization you need to do and greatly reduce any work in future upgrades.

  • If you've overridden community.ftl and want to upgrade it, note that the template has been split into multiple FTL files. This was done to divide logically what was a very large template.

  • Replace WebWork directives with their Struts counterparts. Clearspace version 2 replaces WebWork conventions with Struts. This is pretty much just search-and-replace work to replace @ww. (for WebWork) with @s (for Struts) . The following example shows how to update the url directive. Notice that the updated code also omits the includeParams='none' attribute; in Struts 2, which Clearspace version 2 uses, none is the default value for includeParams.Version 1 (supporting WebWork):

<style type="text/css" media="screen">
    @import "<@ww.url value='/styles/jive-blog.css' includeParams='none'/>";
</style>


Version 2 (supporting Struts):

<style type="text/css" media="screen">
    @import "<@s.url value='/styles/jive-blog.css'/>";
</style>


  • Update calls to APIs that have been consolidated and simplified. Pay attention to the places in a template where code calls methods directly (although, as a best practice, you should avoid calling methods in FTL code and use actions instead). Here's an example in which you'd replace a call to a JiveGlobals methods with a call to JiveResourceResolver:Version 1 (using JiveGlobals ):

<@ww.text name="doc.viewer.more_recent_ver.text">
    <@ww.param><a href="<@ww.url value="${JiveGlobals.getJiveObjectURL(document)}" includeParams="none"/>"></@ww.param>
    <@ww.param></a></@ww.param>
</@ww.text>


Version 2 (using JiveResourceResolver ):

<@s.text name="doc.viewer.more_recent_ver.text">
    <@s.param><a href="<@s.url value="${JiveResourceResolver.getJiveObjectURL(document)}"/>"></@s.param>
    <@s.param></a></@s.param>
</@s.text>


Here's another example where the change was from another class and method, but again to JiveResourceResolver:
Version 1 (using CommunityUtils ):

<a href="<@ww.url value='${CommunityUtils.getCommunityURL(community)}'
    includeParams='none'/>?view=documents" 
    class="jive-link-more"><@ww.text name="doc.main.brdcrmb.documents.link" /></a> 


Version 2 (using JiveResourceResolver ):

<a href="<@s.url value='${JiveResourceResolver.getJiveObjectURL(container)}'/>?view=documents" 
    class="jive-link-more"><@s.text name="doc.main.brdcrmb.documents.link" /></a> 


  • Update community references to container references. One twist on the API changes means that both projects and communities (also known as "spaces") are represented conceptually as containers. In the version 2 API, the Community and Project interfaces both inherit from JiveContainer. To disambiguate between the projects and communities, you'll need to pass a container type with your action calls. The actual disambiguation is handled by Clearspace, however, when it intercepts the call before passing it to the action. The net effect is that the action itself receives only the container ID parameter, not the container type. Here's an FTL example:Version 1 (specifying a community by passing its community ID)

<@ww.action name="community-breadcrumb" executeResult="true" ignoreContextParams="true">
    <@ww.param name="communityID" value="${community.ID?c}" />
</@ww.action>


Version 2 (specifying a community by passing both its type and its ID)

<@s.action name="community-breadcrumb" executeResult="true" ignoreContextParams="true">
    <@s.param name="containerType" value="${container.objectType?c}" />
    <@s.param name="container" value="${container.ID?c}" />
</@s.action>


  • Update code that handles content. This includes code that gets content for display as wiki markup, HTML, and so on. Among the API changes were several designed to streamline and add structure to how you handle content. For example, in version 1, to get content as wiki markup you would have called methods of the message or document or comment itself. In version 2, you pass the content object to a method inherited from JiveActionSupport. These methods include renderToText, renderToHtml, renderSubjectToHtml, and so on. Here's an example using convertToWikiSyntax:Version 1 (using content object method)

<textarea id="comment-body-edit-${comment.ID?c}" name="body" rows="10"
    style="width:100%;font-family:verdana,arial,helvetica,sans-serif;font-size:8pt;"
    class="jive-comment-textarea">${comment.unfilteredBody!?html}</textarea>


Version 2 (using JiveActionSupport method)

<textarea id="comment-body-edit-${comment.ID?c}" name="body" rows="10"
    style="width:100%;font-family:verdana,arial,helvetica,sans-serif;font-size:8pt;"
    class="jive-comment-textarea">${action.convertToWikiSyntax(comment)!?html}</textarea>


 

Removed and Renamed FTL Files

Version 1 FTL File

Version 2 Change

community-document-picker.ftl

Renamed to container-document-picker.ftl

community-thread-picker.ftl

Renamed to container-thread-picker.ftl

import-callback-communitynntp.ftl

Removed.

import-directory-error.ftl

Removed.

import-directory-updatetags.ftl

Removed.

import-directory.ftl

Removed.

datepicker.ftl

Renamed to datetimepicker.ftl

Upgrading Themes

Nearly all of your work upgrading themes will focus on upgrading FreeMarker template (FTL) files. The way you build and deploy themes is unchanged from version 1.

 

But wait -- there's more. Having said that, the best practice recommendation in version 2 is to use widget-customized pages wherever possible as an alternative to custom FTLs. You can use widgets in more places than in version 1; check out the section on upgrading widgets for more.

 

Here's why you should use widgets:

 

  • You can do with widgets much of what you'd do with custom FTL markup. In addition, you can write logic in Java behind the your widget UI.

  • Building widgets to support new features is a good deal tidier than adding new FTL files. While custom FTL files override default FTL files, widgets are encapsulated by the plugin deployment model -- essentially sandboxed.

  • When it's time to upgrade Clearspace, the work needed to upgrade FTL files you've customized would likely be a good deal greater than what's needed to upgrade a widget.

 

Suggested Upgrade Process

Use the following steps as a systematic way to upgrade your themes.

 

  1. Consider which custom FTLs in your themes can be replaced by customizing the page with widgets.

  2. Use jive.devMode Jive property for debugging. By default, Clearspace hides FreeMarker errors in themes. With dev mode on, you'll see them.

  3. Disable themes while upgrading them.

  4. Enable your custom FTL files one at a time. Debugging incrementally will make the process smoother.

 

The information above along with more details can be found in the Upgrading Extensions to 2.0 documentation.

 

We've been noticing more and more discussion on Jivespace about people migrating older customizations to the new Clearspace 2.x architecture along with quite a few people writing new customizations. I thought that now would be a good time for a refresher series of posts on customizing for Clearspace 2.x

 

Clearspace version 2 includes a few big changes to the conventions and frameworks on which Clearspace is built. Many of these changes were made to improve Clearspace extensibility by letting the application and extensions be more loosely coupled -- and so be more durable during upgrade. Other changes that impact customizations were simply feature improvements in which some change to code is required.

 

Here's a high-level list of the changes that effect extension and customization code:

 

  • Clearspace was migrated to the Spring framework. This has both broad and deep effects on code written to run on Clearspace; such code must use the same conventions in order to integrate well. The changes include API refactoring to support dependency injection and integration of Spring modules (for security and transaction management, for example).

  • Clearspace was migrated from WebWork2 to Struts2. This has broad impact on plugins, but the changes are relatively small  -- primarily effecting syntax in FreeMarker templates and configuration files.

  • Widgets were enhanced to support use in multiple contexts, rather than just space overview pages. Changes to widget development support this new feature.

  • The macro-related API was narrowed to the area documented in version 1, essentially excluding an undocumented area that some people used. Also, macros must now return HTML that qualifies as well-formed XML.

  • Web services support is now provided through CXF, an evolution of XFire (the version 1 technology). A larger change results from the migration to Spring, which makes integrating new web services a bit more complex.

  • Custom user data providers are now based on a streamlined API. These must be rewritten, but the new model is much simpler.

  • Custom authentication providers are now based on Spring through Acegi. The change is pretty significant, including new API and configuration conventions.

  • Myriad API changes resulted from the migration to Spring and from an effort to streamline and modularize the Clearspace API. These include support for the projects feature, refactoring manager interfaces, content handling, and conventions such as dependency injection.

 

For those extensions and customizations deployed as plugins (widgets, actions, macros, and web services), there are a few changes in version 2 with broad effect.

 

  • Migration from WebWork to Struts means replacing your xwork-plugin.xml with a struts.xml

  • Migration to Spring means adding a spring.xml for certain kinds of Spring support.

  • One benefit of the new model is that modifying your plugin.xml, struts.xml, or spring.xml will provoke Clearspace to reload your plugin. This can be handy for debugging.

  • Support for a <maxserverversion> element was added. Similar to the <minserverversion> element, use the new one to specify that highest Clearspace version on which your plugin is supported. This can be useful when you want to ensure that upgrades to Clearspace prompt the plugin's users to upgrade the plugin also.

  • Plugin life cycle methods changed. If you used the version 1 com.jivesoftware.base.Plugin interface, you'll notice that its two methods have changed in order to support Spring. Semantics for the methods is unchanged.

    • Plugin.initialize(PluginManager, PluginMetaData) is now Plugin.init, a method without parameters. Use Spring dependency injection to receive a PluginManager instance; you can retrieve a PluginMetaData instance from the manager.

    • Plugin.destroyPlugin is now Plugin.destroy.

 

The information above along with more details can be found in the Upgrading Extensions to 2.0 documentation.

 

 

Clearing or containing elements with class="clearfix"

I've had more than one engineer ask, "what is the CSS class 'clearfix'?"

Short answer:

It's an easy way to make sure floated content is contained by its container (when you want it to). It's cross-browser friendly and it prevents the need to start floating everything and their parents which can lead to other issues. See the attached file for a live example.

 

You don't need to go crazy and start adding class="clearfix" to everything. Just add it only to parent elements that need to contain floats where content coming after but sibling to that parent needs to clear the floats.

Long answer:

Floats are meant to break out of their parent container and ancestors on purpose, allowing for text or other sibling or descendent content to wrap something like an image. But as is often the case, UI designers want subsequent content to clear that float for the purpose of controlled layout and predictability. There are many ways to achieve this:

  • Adding extraneous elements like <div style="clear:both"> </div> or an HR or BR with styles applied. Drawbacks: code bloat; non-semantic markup; easy to disassociate with the floated element in the markup.

  • Add clear:left; to the next element (sibling to the float or sibling to the float's parent). Drawbacks: that will make that element clear all floats (including ones you don't want it to); can easily be disassociated with the original float context in the markup.

  • Start floating everything. Drawbacks: This creates more layer-caked problems; it's compounded by IE on Windows (up until, but not including IE8) with its hasLayout rendering property (see technical answer).

  • Rely on a class name with browser targeted properties in CSS fix. This is our rock star …

Technical answer (see attachment):

Floats, as mentioned, are meant to break out of their parent containers. When we want a subsequent item to clear that float, what we really want is the float or floats to be contained by its parent element and not break out. With one class="clearfix" on a parent element (the CSS definitions for which have been added to jive-global.css as of 2.0) we can achieve that.

 

For compliant browsers (any modern browser that is not IE, and IE8), we do that with generated content, this is future proof as it's also standards compliant. For IE on Windows (below IE8) we do that with targeted CSS hacks that are ignored by modern browsers, giving it the mysterious, completely non-compliant property of hasLayout. The mumbo jumbo black magic of hasLayout is just nasty nasty and nasty. It's responsible for or related to all kinds of bugs in IE Windows. For the best explanation in existence on hasLayout, see http://www.satzansatz.de/cssd/onhavinglayout.html. We can even target the other dead IE, IE mac, and give it its own CSS fix; but it's not really needed for CS support.

 

Here's the CSS definitions:

* for compliant browsers */
.clearfix:after {
    clear: both; 
    content:".";
    display: block;
    height: 0;
    visibility: hidden
    }
/* affects only IE7 */
.clearfix {
    min-width: 0;
    }
/* targets only IE5-6 and hidden from lowly IEmac \*/
    * html .clearfix { height: 1%; } 
/* end hide IEmac */
/* What? You want to get crazy and do IEmac too? Okay. \*//*/
    .clearfix { display: inline-table; } 
/* end IEmac only */

 

EDIT: I realized I never attached the example HTML file.

 

Learn more about using web services to access your data in Clearspace 2.0 with Andrew Wright, Jive Software Engineer.

 

 

 

You can also download the Quicktime version (Caution: file is ~232MB), or you can watch a larger version online, which will improve readability of embedded screenshots (recommended).

 

The entire presentation is also attached below as a PDF file.

During the last couple weeks of the 2.0 development cycle we pushed some really helpful search improvements (some of them bug fixes) into Clearspace. There are a numberof posts scattered around our intranet (which is called Brewspace) where the actual improvements / bug fixes are discussed but I don't believe those improvements made it into the documentation (there are a number of improvements listed in the changelog, but no description of what the improvements are). Hence this blog post.

 

First, in 2.0, the default search operator was changed to 'AND' from 'OR', the end result being that if you did a search like this:

clearspace openid

Clearspace would look for all the blog posts, discussions and documents that contained the term "clearspace" AND contained the term "openid". Way back in Clearspace 1.0 the thought was that we should deviate from what Google does (they AND the terms you input) because we're not searching the entire web; our thinking was that most of our installations would only have a couple thousand documents, blog posts and threads and so we didn't ever want a search for 'clearspace openid ldap' to return nothing if there was a document that discussed two of the three. The reality is that when the search operator was 'OR' the number of results from a search query in Clearspace was almost always greater than 500 results (the maximum number of results we would return in a search): in fact, the more words you used in your query, the more likely that you'd end up with a large number of results, which in theory is great (we found a bunch of stuff that for you!) but in practice doesn't make for a great user experience (thirty four pages of search results? come on!). One of the articles (discussed below) had this to say about lots of search results:

Users sometimes measure the success of a query primarily by the number of results it returns. If they feel the number is too large, they add more terms in an effort to bring back a more manageable set.

So not only did "OR" result in more results per query, if you decided that you wanted to refine your query by adding a term, the number of results would actually grow, not shrink, which is the opposite of what you'd expect.

 

The funny thing about changing this default is that when we turned it on for brewspace (our intranet, no other changes had been made at that point), a number of people noticed right away and were amazed at the 'improvement'. It's really crazy how something as simple as "AND" versus "OR" could make such a big difference in user experience.

 

Before I move on, here are a couple interesting articles I found that talk about search as it relates to user experience:

  • Greg Linden, an ex-Amazon guy, wrote a great blog post a couple months agothat summarized a talk that Marissa Mayer gave about Google and their results page and the number of results per page and also added some notes about his own experience while working at Amazon. The bottom line from the presentation? Speed kills. The faster that we can return search results, the happier Clearspace users will be (although the comments on that post tell a potentially different story, don't miss'em).

  • A BBC News article from 2006 had this to say about search results:

At most, people will go through three pages of results before giving up, found the survey by Jupiter Research and marketing firm iProspect. It also found that a third of users linked companies in the first page of results with top brands. It also found 62% of those surveyed clicked on a result on the first page, up from 48% in 2002. Some 90% of consumers clicked on a link in these pages, up from 81% in 2002. And 41% of consumers changed engines or their search term if they did not find what they were searching for on the first page.

Takeaway? Relevant results are more important than many results.

The essential problem of search — too many irrelevant results — has not gone away.

More and more, our ongoing research is telling us that Search has to be perfect. Users expect it to "just work" the first time, every time.

 

One thing that was easy to add which has also come up a couple times was search by author. I'm happy to report that in 2.1 we added the ability to search for content authored by a specific user. So just like you can click 'more options' on the search results page today and choose what types of content you want to search for, you'll be able to select a user whose content you want to find and filter the results using that selection.  Side note: that functionality has always been in our API and if you're a URL hacker like I am, you can actually perform Clearspace searches using a pretty URL like this:

http://example.com/clearspace/search/openID

or if you want to search for any content written by the user 'aaron' that contains the word 'openID', you'd use this URL:

http://example.com/clearspace/search/~aaron/openID

 

Another thing that I believe has worked in the past but that we haven't talked about is the idea of a simple syntax for search. Much like the operators you can use in Google (ie: 'site:jivesoftware.com lucene' will find all the references to 'lucene' on the domain 'jivesoftware.com'), we now support the following operators: 'subject:', 'body:', 'tags:' and 'attachmentstext:'. While I admit that they're not the most user-friendly things to type in, it does give advanced users a little bit more flexibility. For example: you can now ignore tags if you want by doing a search like this: 'subject:lucene OR body:lucene'. The search syntax operators are scoped to be in the search tips documentation that sits right alongside the search box. Again, this is for 2.1.

 

Those were the improvements. Now the bug fixes (which just so happen to also really improve your searching experience). 

  • Search stemming doesn't seem to be working (CS-3645): Not sure how long this wasn't working, but the existence of this bug meant that if you put the word "error" in a document and then did a search for "errors", our search engine wouldn't find your document. Read more about stemming if you're curious about that sort of thing. If you're seeing this bug, make sure a) you upgrade to at least Clearspace 2.x and b) make sure that you're using a stemming indexer. The default analyzer does not stem. You can change the indexer by going to the admin console --> system --> settings --> search --> search settings tab --> indexer type.

  • Group search results by thread setting in admin console doesn't change search behavior (CS-3656): I'm not sure how long this feature has been around, but there is a search setting in the admin console that gives you the ability to group all messages in a thread into one result in a search results page so that the messages in a single thread don't overwhelm the search results (since messages share the subject and tags from the thread, that actually happened quite a bit). Fixed in 2.0, I highly recommend turning it on in your instance if you haven't already.

  • Search updates to better balance queries across content types (CS-3638): Some improvements were made in 2.0.0 toward this issue, but it's 100% fixed in 2.0.3 and 2.1. There were two really big but really really hard to see problems with the way that we were executing our search queries. First, a quick background on how a search query is performed in Clearspace against all content types. We have a single Lucene index for all the content in Clearspace (there is a separate index for user data, but that's a different story) so when a search for 'bananas' is executed, we did something like this (don't read too much into the language I'm using, I'm just trying to illustrate how it works at a 30,000 foot level):

  1. get blog posts that match query

    • find all the blog posts where the subject matches 'bananas' OR the body matches 'bananas' OR the tags matches 'bananas' OR the attachments matches 'bananas' or the blogID matches 'bananas'

  2. get discussions that match query

    • find all the messages where the subject matches 'bananas' OR the body matches 'bananas' OR the tags matches 'bananas' OR the attachments matches 'bananas' or the threadID matches 'bananas'

  3. get documents that match query

    • find all the documents where the subject matches 'bananas' OR the body matches 'bananas' OR the summary matches 'bananas' OR the fieldsText matches 'bananas' OR the tags matches 'bananas' OR the attachments matches 'bananas' or the documentID matches 'bananas'

  4. merge results from steps 1-3 using the relevance score from each item in the result set as the comparator

  5. display results

The assumption we made when writing this code was that the scores that Lucene returns for each item of all content type will be relatively similar. More concisely, if I had a document and a blog post which for some reason had identical content, I'd expect they would both have the exact same Lucene relevance score if they came up in the results of a search. But that assumption turned out to be wrong, not once but twice.

 

First, as you can see from glancing at the sample queries I pasted above, we searched a different number of fields per content type. Who cares right? Why would the number fields that you search on influence anything? Turns out that Lucene cares: the way scoring works in Lucene is that if you search on ten fields and only get a hit on two of them in document 'X' that the resulting relevance score should be less than a hit on blog post 'Y' where you search on five fields and get a hit on two of them. It makes perfect sense when you think about it: it's just like the tests you had in school. Getting a 4 out of 5 on a test works out to be 80%, about a B. If you got 4 out of 10 on a test, that's 40%, you failed. You probably called them grades... maybe sometimes even a score, which just so happens to be exactly how Lucene refers to the relevance that a particular document has to a given query (if you're curious about how Lucene does scoring / relevance you should check out the JavaDoc for the Similarity class and also read this document on scoring). Anyway, this behavior was actually fixed in 2.0: so now when we execute a search on mixed content we search the exact same number of fields for each content type: subject, body, tags, attachmentsText.

 

The second assumption that turned out to be wrong was just as nebulous. I illustrated above how we search in Clearspace: we do searches for each content type and then we merge the results of those searches into a single result set. In order to do a query for just blog posts that match the word 'token', we do a query that in Lucene looks like this:

+objectType:38 +(subject:token body:token attachmentsText:token tags:token)

It kinds of looks like a SQL subselect: get me all the things where one of subject, body tags or attachmentsText match and then, from those results, only return the results where the objectType is 38 (which is the int that JiveConstants.BLOGPOST refers too). The thing that killed us here was outer statement

+objectType:38

because:

a) when Lucene executes a query, it computes a query weight and field weight for each statement in your query and multiplies those two values together to get the total weight for that statement and

b) the query weight is basically a measure of how many times the key (in this case 'objectType') appears divided by the number of times the value appears (in this case '38') which means that

c) content objects that you have less of (in our case: blog posts) will tend to have a score much higher than content objects that you have a lot of (in our case: documents). Again, this makes sense: items that appears in the index a relatively small number of times are in some sense rare and so they should get a relatively higher weight. Regardless, it turns out there's a really easy fix for this problem as well: you can boost specific fields in your query like this:

subject:token^3

and you can effectively neuter a field by boosting it to zero:

subject:token^0

which means that Lucene will look for all the items in the index whose subject is 'token' but the weight, which usually influences the score that it assigns to the field 'subject' will not influence the score that the resulting item receives. 

 

We're continually looking to improve the search tools in Clearspace. If you're seeing something you don't expect or if there's something cool you'd like us to add, please pipe up in our Support and Feature Discussion spaces here on Jivespace.

 

 

We just launched our new Jivespace Plugin Directory for Clearspace plugins! You can now more easily find and download plugins from Jivespace, and if you are developing plugins, you can add your plugins to the new directory.

 

Plugin Directory

 

The really cool part is that we created the entire plugin directory as a plugin to Clearspace. Our web engineer, Tim, took our existing document content type and extended it to create a new plugin type with additional information relevant to plugins along with a new look and feel for the plugin directory.  He also used the plugin jar to pull almost all of the metadata displayed with the plugins, including license, logo, readme, compatible versions, and much more. For developers, this means that you only need to enter the information in your plugin, instead of having to duplicate all of the information by filling out redundant forms. It also means that when you update your plugin jar file with a new release, the plugin information will be automatically updated.

 

Tim is currently working on polishing the code a bit, but he will be releasing this as a plugin for other people to use as a plugin marketplace. It also provides a very useful example of how to extend an existing Clearspace content type to create a new content type in Clearspace.

 

Keep in mind that this is just the first version of our Plugin Directory, and we plan to start making incremental improvements and enhancements. But first, we want to hear from you. Take a look at the new Jivespace Plugin Directory. What do you like? dislike? How can we make this even better in future revisions?  Please leave comments with your ideas on this blog post!

 

 

 

 

Learn all about how to write new widgets for Clearspace 2.0 from Aaron Johnson, Engineering Manager at Jive.

 

 

 

You can also download the Quicktime version (Caution: file is ~285MB), or you can watch a larger version online, which will improve readability of embedded screenshots (recommended).

 

The entire presentation is also attached below as a PDF file.

 

You can also watch an earlier video about developing widgets for Clearspace 2.0

This is the second blog post about real time communication support in Clearspace that started with Connecting a chat client to Clearspace. Today we are going to cover chat events that are going to be available in Clearspace 2.1. Lets start giving some personal examples to illustrate real usages for chat events.

 

  1. Every Wednesday at 10:00 AM PST developers of igniterealtime.org projects join the chat service to answer development questions. Members of the community can join the chat service using their XMPP client of choice or using the web client. Moreover, users of other XMPP servers can also join the chat service as long as server-to-server is enabled.

  2. Every Monday morning developers of the Real Time Communication team, that includes local and remote developers, join a chat room to discuss status updates and goals for the week.

  3. Last week I was invited to a meeting to discuss some technical problem about clustering.

 

In the above examples we see different usages for chat events. The first example is showing a repetitive chat event that is not associated to any space, project or social group but to the entire site. In the second example we have a project, but it could be a space or social group, whose members meet every week to discuss their work. As you would expect, permissions of the container are applied to determine who can join the group chat. And finally, we see that scheduled chats could be a one time only activity.

 

A chat transcript is created for each occurrence of the chat event. When the event is over the transcript is "closed" and moderators can moderate/edit it. As any content in Clearspace, transcripts are indexed on real time and can be searched just like you search for a document or a blog.

 

A persistent room in Openfire is created for each chat event. A new groupchat service was added to Openfire that interacts with Clearspace to control who can create, join, configure and delete rooms. As I said in my previous blog, it is possible to use your own XMPP client to join a chat event or you can just chat from the Clearspace site. File transfers in rooms was also added to Openfire but the Clearspace side is not ready yet so the feature will be ready for the next release of Clearspace. However, since you can use your own XMPP client it is still possible to make use of the new functionality to share files in a room. The file transfer feature is based on WebDAV File Transfers

 

Next week we will cover other type of conversations that are going to be part of Clearspace 2.1.

 

 

 

Filter Blog

By date: By tag: