Getting Started with the External Storage Framework

Version 53

    NOTE! This document is a work-in-progress. It still has some sections that are not complete. We intend to add more details soon!

                                              

     

    Overview

     

    External Storage Framework (ESF) is a powerful solution for connecting a Jive place to an external storage system. To the user, there is seamless integration between systems: if a file is uploaded via Jive, it will appear on the external system; if a file is uploaded via the external system, it will appear within Jive. Under the hood, all binary data is stored within this external system. However, this content is treated like a first-class content type within Jive: you can edit it, you can search for it, you can comment on it. Commenting on files also works bidirectionally, if desired. Existing ESF solutions include Box, Google Drive, Dropbox, and others. You can easily create a new ESF with the Jive Platform. When providing a new ESF, you must create an external storage provider (ESP), which is a REST service between Jive and the external storage system.

     

    Note: You can bake the ESP into plugins, for example, the Sharepoint add-on uses the ESP located in the Extended API's plugin.  Effectively you can use plugins to create and house your REST service in the core application database.

    esf-arch.jpg

     

    Basic Concepts

     

    Installation & Registration

     

    An ESF is installed as an add-on to a Jive community. It must be installed by a Community Manager. Once installed and configured, a user can create a place and use your ESF as the storage mechanism for that place. After setting up a place and associating it with your ESF, Jive registers with your ESP and your ESP responds with additional endpoints that Jive calls when events occur.

    esf-seq0.jpg

     

     

    Uploading & Downloading with Jive

     

    When a user uploads or downloads a file from a Jive place associated with your ESF, Jive will communicate with your ESP endpoints in order to transfer the file. No binary data is stored inside of Jive. Your ESP is responsible for managing this data. Your ESP also provides Jive with additional ESP endpoints so that you will be notified when changes to this file occur.

    esf-seq1.jpg

     

     

    Uploading & Downloading with External System

     

    Depending on the external storage system you are connecting to, a user may attempt to upload or download the file using the third party system (i.e. outside of Jive). If the user uploads or modifies a file, your ESP will need to perform a Jive ESF REST API call (e.g. POST /{Container ID}/files) in order to let Jive know that a file has been added or modified.

    esf-seq2.jpg

     

    Commenting on Files

     

    Similar to how uploading a file works, when a comment is made on a file within Jive (such as liking or commenting on content), your ESP endpoint is notified; and when activity occurs from the third-party system, it is your responsibility to notify Jive using a Jive ESF REST API call (e.g. POST /files/{ESP File ID}/comments).

    esf-seq3.jpg

     

    Deleting or Modifying a File

     

    Just like when uploading a file, when a file is modified or deleted from within Jive, Jive calls your ESP. From there, you can update the external storage system as necessary. And when a file is modified or deleted from outside of Jive, you should call the appropriate Jive ESF REST APIs.

     

    Example (using the Jive Node SDK)

     

    The Jive Node SDK comes with a simple ESF example that uses a local file system as the external storage system. This section describes how to run the example, and then talks about how it is implemented.

     

    Before You Begin

     

    1. Install the Jive Node SDK on your development system. Installation instructions are described at Getting Started > Installing the Jive Node SDK.
    2. You need to be a community manager or administrator on the Jive server being used for this tutorial. (You will not be able to use the Jive Sandbox for this example because most users do not have the appropriate rights to install an ESF.)
    3. Your development system must be accessible to the Jive server. If your system is not accessible to the Jive server (e.g. behind a firewall), consider using a cloud-based IDE for this tutorial. See Using Nitrous.IO with the Jive Node SDK for one type of cloud-based solution.

     

    Instructions

     

    1. Browse to an empty directory where you want your code for this tutorial to reside.
    2. Install the example using the Jive Node SDK.
      jive-sdk create esf
      This creates a project with all the code necessary to build a simple external storage provider that uses the local file system as the storage system.
    3. Update all node package dependencies.
      npm update
    4. Start your integration service (ESP) and upload the add-on using the following instructions: Getting Started > Deploying a Jive Node SDK Project as an Add-On.
    5. Select the Storage Management tab. (If you do not see this tab at first, refresh the page.) When you're at this tab, click Add Integration.
      esf-dialog.png
    6. Add the new ESF integration. Select the provider name (this is defined by the "displayName" value in your definition.json file and is "File System Storage" by default). Pick a display name that user's will see when they configure an ESF.
      esp-dialog2.png
    7. Save the new ESF integration.
      esf-dialog4.png
    8. Create a new group. Click Preview.
      esf-dialog3.png
    9. Before saving the group, change the External file storage.
      esf-dialog5.png
    10. Select your new external storage provider, which uses the name you specified in a previous step.
      esf-dialog6.png
    11. Click the Create Group.

     

    Testing the ESF

     

    You are now ready to test out your new file-based ESF:

     

    1. Upload a file to the group.
      esf-dialog7.png
    2. Note that all files for this group are stored within the JiveStorage/{Group Name} folder wherever your service is running. They are versioned so that you can retrieve older versions of a file.
    3. From within Jive, you can modify or delete the file, and these changes are reflected by your external storage system.
    4. This service also supports bidirectional file syncing. If you inspect the storage folder on your system, you'll see that each group contains an "uploads" folder. Put a file inside that folder. Within five seconds, that file will disappear and a new folder will be created for that file. If you browse to your Jive group, you'll see that new file available for download.

     

    How It Works

     

    Three files make up the main part of this ESF example:

     

    • ESF Definition (definition.json). The /storages/fsstorage/definition.json file contains the ESF definition that is bundled with the add-on. The contents of this file are automatically incorporated (by the Jive Node SDK) into the "storageDefinitions" array of the add-on's defintion.json. Among other things, it describes the registration URL that Jive calls when you configure a place to use your ESP. Refer to Add-on definition.json below for more details.
    • ESP Routes (explicit_routes.js). The routes for your service are located at /storages/fsstorage/backend/routes/explicit_routes.js. In this file, you will see how Node.js sets up the routes for the following endpoints: isAlive, registerPlace, uploadFile, uploadVersion, downloadVersion, deleteContainer, deleteFile. Some of these routes are given to Jive at registration time. Other routes are given to Jive in response to another Jive request. For instance, when Jive notifies your ESP about a new file, your ESP responds with information about where to call when the file is modified or deleted:

      var responseData = {
          'externalId': fileGuid,
          'version': version,
          'resources': [
              {
                  'name': 'self',
                  'url': '/fsstorage/file?file=' + fileGuid,
                  'verbs': ['GET', 'PUT', 'DELETE']
              },
              {
                  'name': 'uploadVersion',
                  'url': '/fsstorage/uploadVersion?file=' + fileGuid,
                  'verbs': ['POST']
              }
          ]
      };
      ...
      response.send(responseData, 200);
      ...
              

      To understand more about how Jive knows to call these routes, refer to ESP Endpoints below.
    • ESP Implementation (storage_service.js). The actual implementation for the ESP endpoints can be found in /storages/fsstorage/backend/storage_service.js. These functions illustrate how the ESP manages the files associated with a Jive place. This file also contains the function (checkUploadFolder) that checks the upload folder and pushes new updates to Jive using the ESF REST API (i.e. POST .../{Container ID}/files). Refer to Jive ESF REST API below for more details.

     

    Further Details

     

    Add-on definition.json

     

    This file is one of the essential elements of the add-on package. Among other pieces of information, it contains the following properties that are related to the ESF:

     

    Property NameDescriptionValue Type
    displayNameThe display name, as seen when adding the External Storage Provider integration to a place.String
    registerPlaceThe URL representing the ESP endpoint that Jive calls when a user registers a place to work with your ESP.URL
    userMappingOptions

    Defines how users are associated to the files in the external storage system.

    TO DO: Explain this more thoroughly...

    Array with one or more of the following elements: EMAIL, USERNAME,

    TO DO: List all options

    supportedResourceNames

    The type of features your ESP supports.

    TO DO: Explain this more thoroughly.

    Array with one or more of the following elements: uploadFile, downloadFile,

    TO DO: List all options

    healthCheckURLThe URL that Jive calls to ensure that your ESP is operational.URL

     

    Sample definition.json:

     

    {
        "storageDefinitions": [
            {
                "displayName": "File System Storage",
                "description": "Saves files on the node.js machine",
                "registerPlace": "%serviceURL%/fsstorage/register",
                "userMappingOptions": [
                    "EMAIL"
                ],
                "supportedResourceNames": [
                    "uploadFile",
                    "downloadFile"
                ],
                "prefixUrl": "%serviceURL%",
                "resolveResources": "%serviceURL%/resolveResources",
                "icons": {
                    "16": "extension-16.png",
                    "48": "extension-48.png",
                    "128": "extension-128.png"
                },
                "healthCheckURL": "%serviceURL%/fsstorage/isAlive",
                "name": "38f231d5-4883-5164-88cf-7c187950bc54"
            }
        ]
    }
                                    

     

    ESP Endpoints


    Jive needs to know about your ESP endpoints in order for it to call you when activity occurs. The following table describes how Jive knows about your various endpoints:

     

    Endpoint FunctionalityHow Jive knows to call this endpoint

    General Endpoints

    Health checkThe "healthCheckURL" property in definition.json. Refer to the "StorageDefinitionData" page from the ESF Documentation for additional information.
    Place / Container Endpoints
    Register a placeThe "registerPlace" property in definition.json. Refer to the "StorageDefinitionData" page from the ESF Documentation for additional information.
    Delete a placeThe "Register a place" endpoint responds to Jive with this endpoint as a resource. Refer to the Resources section within the "ExStorageContainerRegistrationResponse" page from the ESF Documentation for additional information.
    File-specific Endpoints
    Upload a new fileThe "Register a place" endpoint responds to Jive with this endpoint as a resource. Refer to the Resources section within the "ExStorageContainerRegistrationResponse" page from the ESF Documentation for additional information.
    Upload a new file versionThe "Upload a new file" endpoint responds to Jive with this endpoint as a resource. Refer to the Resources section within the "ExStorageFile" page from the ESF Documentation for additional information.
    Download a file versionThe "Upload a new file" endpoint responds to Jive with this endpoint as a resource. Refer to the Resources section within the "ExStorageFile" page from the ESF Documentation for additional information.
    Delete a fileThe "Upload a new file" endpoint responds to Jive with this endpoint as a resource. Refer to the Resources section within the "ExStorageFile" page from the ESF Documentation for additional information.
    Add a comment to a fileThe "Upload a new file" endpoint responds to Jive with this endpoint as a resource. Refer to the Resources section within the "ExStorageFile" page from the ESF Documentation for additional information.

     

    Jive ESF REST API

     

    The ESF REST API is what you use to tell Jive about changes to the external storage system that occur outside of Jive. If someone uploads a new file, modifies a file, deletes a file, comments on a file, you use an ESF REST API endpoint to tell Jive. Some example endpoints include:

     

    REST EndpointDescriptionExample URL
    /containers/{Container ID}/filesPOST to this endpoint to create a new file.http://my.jive.instance/api/core/v3/exstorage/containers/1047/files
    /files/{File ID}/commentsPOST to this endpoint to add a comment to a file.http://my.jive.instance/api/core/v3/exstorage/files/5827/comments
    /files/{File ID}/trashPOST to this endpoint to delete a file.http://my.jive.instance/api/core/v3/exstorage/files/5827/trash

     

    For details about all the ESF REST API endpoints, refer to: External Storage Framework - Documentation.

     

    Reference Documentation

     

    Refer to the ESF documentation: External Storage Framework - Documentation.