Technical Preparations for a Move to Jive Cloud

Version 10

    Overview

    The article walks you through the concepts and steps involved in transferring your Jive databases and binary storage to Jive's data center for a move from an on-prem Jive installation to Jive Cloud.

     

     

    Initially, the customer and Jive Professional Services work together to prepare the system for the move, which–among other steps–includes binary storage (binstore) preparations and secure transfer account provisioning. It may also include a conversion of the source database to PostgreSQL, if necessary. Subsequently, the following steps will be tested one or more times during a dry run, before the actual cutover:

    1. If the instance's binary data is currently stored in the application database and its total size is larger than 3 GB, the customer must first convert to file system storage. See below.
    2. Customer prepares database and binstore backups.
    3. Customer transfers the backups to a secure server provisioned by Jive.
    4. Jive Hosting staff picks up the backups to transfer them to the target Jive instance.
    5. Jive Hosting then restores the database.
    6. If the binstore is in the Jive database, which is acceptable for binstore sizes less than 3 GB, Jive Hosting staff converts the binstore to be the file system-based one instead.
    7. The upgrade procedure from the current to the target Jive Hosted or Cloud version of Jive is executed.
    8. The instance is handed over to the customer for smoke testing.

     

    Steps & Concepts in Detail

     

    Binary Storage (binstore)

    A Jive Storage Provider implements an approach to store binary data uploaded by your users to Jive. Out of the box, Jive supports two local Storage Provider configurations:

    1. FileStorageProvider, which stores all binary content in a set of deeply nested folders in a specific file system location.
    2. JdbcStorageProvider, storing the binary content in the jivebinstore table in your Jive instance's application database.

    You can determine the currently configured provider for your Jive instance by going to Admin Console > System > Settings > Storage Provider. There, you will also be shown the number of Total Bytes Stored in the binstore.

     

    Data Transfer and Cutover Duration

    You will be transferring your Jive database backups and your binstore to a dedicated secure file server in Jive's data center. The database backup must be transferred for every dry run of the upgrade, as well as for the actual go-live. For file system-based binstores, an incremental strategy is described below. We have observed widely varying data transfer rates in the past and they usually fall in the range of 1 to 4 MB/sec. This means that transferring–for example–100 GB of backups/binstore archive can take up to 30 hours each time, including during production production cutover.

     

    The following table illustrates the impact of the size of the data to be transferred, specifically the databases, on the overall dry run/cutover timing. The actual values given are merely a rough estimate for duration:

    Step1 GB10 GB100 GB
    Back up source databasesa few minutesup to half an hourup to 5 hours
    Copy source database to Jive secure file serverless than half an hourup to 3 hoursup to 30 hours
    Restore database backups in Jive Data Centera few minutesup to half an hourup to 5 hours
    Convert Storage Provider to File System*, if necessarya couple minutesup to half an hourup to 8 hours
    Total1/2 to 1 hour4 to 5 hours40 to 60 hours

    *Because of the excessive time necessary to handle large databases, Jive requires converting existing database binstores to file system ones. This enables alternative transfer strategies for the binstore.

     

    Data Restore at Jive's Data Center

    Jive Hosting staff will pick up the backup files from the secure file server and transfer them to your new Jive instance, where the database backup will be restored. For large databases, this can take several hours, so to minimize cutover time, we need to minimize the size of the databases. If the JdbcStorageProvider is configured for your instance, Jive Hosting will also perform the migration to the FileStorageProvider, which will take an additional several hours, depending on the size.

     

    Changing the JDBC Storage Provider to File System

    Please read and understand the following documentation for your Jive version before continuing with the conversion:

    Jive 6 - Configuring a Binary Storage Provider

    Jive 7 - Configuring a Binary Storage Provider

    Jive 8 - Configuring a Binary Storage Provider

    JIve 9 - Configuring a Binary Storage Provider

     

    The basic procedure for converting the storage provider is as follows:

    1. Plan for the duration of the conversion. A good starting point to estimate is that each 1 GB of binstore will take roughly 2 minutes to convert. The actual conversation duration greatly depends on the throughput of multiple components, including your database's and the target file system's storage backend. Make sure both are tuned optimally. Converting a 100 GB binstore can take anywhere from 2 to 8 hours.
    2. Ensure that the conversion happens during low-traffic time on your Jive instance. While your Jive instance will remain available to users during the conversion, including continuing to support binary file uploads, the conversion puts additional load on your Jive instance and may degrade user experience.
    3. Ensure that the target file system volume is sufficiently sized and has high throughput performance. We recommend a target volume that is at least 25% larger than your current binstore. Add capacity according to your instance's past binstore growth to ensure sufficient available space for new uploads performed between the time of the storage provider conversion and the actual cutover to Jive's data center.
    4. Note that the file system will need to be mounted and available to all Jive web application nodes running on your instance
    5. Perform a full backup of your Jive application database, in case a roll-back is needed.
    6. When ready to perform the conversion, go to the Storage Provider page in the Jive admin console and select:
      Screen Shot 2014-10-02 at 9.37.27 AM.png
    7. From there, select the FileStorageProvider. In the next step, leave the Namespace as is and provide the full absolute path to the new binstore root folder on the volume provisioned earlier. Continue.
    8. Make sure to not select the option to delete the binstore files in the original provider:
      Screen Shot 2014-10-02 at 7.46.16 AM.png
    9. Execute and monitor the conversion.
    10. After the conversion is complete, edit the cache settings on the same page and disable the cache altogether.
    11. Test the new binstore by uploading a couple binary documents and redownloading them, verifying that the jivebinstore table is not being modified by these operations.
    12. Once you have verified that the new storage provider operates satisfactorily, drop the entire jivebinstore table from your Jive application database.

     

    Picking a Data Transfer Strategy

     

    Overview

    1. Customer: Provide fixed corporate IP addresses from which you will connect to Jive's data center.
    2. Customer: Pick SFTP or RSYNC.
    3. Jive PS: Provision secure file server drop account.
    4. Customer: Send initial database backup and binstore export.
    5. Customer: Send differential binstore additions for subsequent dry runs and the go-live.

     

    Jive Professional Services offers two strategies for transferring your data to Jive's Data Center: SFTP and RSYNC. Please read the instructions below to determine which one is appropriate for your team. For either to be provisioned by Jive, you will need to provide one or two fixed corporate IP addresses from which you will establish secure data transfer connections to our data center. These IP addresses will be whitelisted on Jive's firewall to allow connections from them. Often, such corporate IP addresses will also need to be whitelisted by your IT staff to allow outgoing connections on port 22.

     

    A customer-specific drop account and folder will be provisioned for you in time for the transfer.

     

    SFTP

    With the SFTP credentials provided to you at a later point, you may either connect via an SFTP client or use SCP. Database backup files can be transferred to the drop folder without further processing. The file system binstore, however, needs to be compressed into a single archive to transfer efficiently. We recommend an incremental archiving strategy to minimize transfer times during subsequent dry runs and the actual cutover. The following is an example using the *nix TAR command, but a similar strategy can be implemented using ZIP and the appropriate command line parameters:

    1. For the initial transfer–dry run 1–compress your entire binstore. We recommend making the date of the backup part of archive name*:
      tar -cvzf binstore_20140830.tar <path to binstore>
    2. For all subsequent transfers, only archive the files that have been changed or created since the last transfer. We recommend taking the last archive's timestamp and subtracting a day to provide an overlap of dates:
      tar -cvzf binstore_20140914.tar --after-date=2014-08-29 <path to binstore>

    After each backup and archive creation, also generate MD5 checksums for all files to be transferred. As part of the dry run communication, you will send them to us and we use them to validate proper transfer to your Jive instance.

     

    *If your binstore does not contain a large fraction of compressible files, you may chose to not use compression by dropping the "z" from the TAR command, to reduce archiving time. This may, however, result in increased transfer times. You may want to test the differences in archive sizes as part of your dry run 1 preparations.

     

    RSYNC

    RSYNC simplifies several of the steps that would otherwise be necessary for SFTP-based transfers.

    1. Because RSYNC relies on checksums for its differential transfer, a separate calculation of checksums is not necessary.
    2. RSYNC can efficiently transfer deep folder structures, so archiving the binstore into single file is not necessary.
    3. Incremental binstore archiving is also not necessary.

    Overall, RSYNC is the more robust and simpler strategy to implement, but care must be taken to set up the command's folder specifications correctly. Because of it's advantages, the sync command provided below can simply be scheduled via a cron job, continuously keeping Jive updated with your binstore.

     

    Jive will provide the secure server host name and a private key file (id_dsa) to authenticate. The file server is listening on standard ports for rsync over ssh.

     

    Please note that all information provided below assumes a Linux-like system from which you will be executing the binstore synchronization.

    1. Download the provided private key file to the machine from which you will be running rsync.
    2. Unzip it and make sure it has the proper file permissions to be used by ssh: $ chmod 400 id_dsa
    3. Execute rsync as follows, using actual paths on your system:
      $ rsync -av --delete --progress --partial -e 'ssh -l customer -i ./id_dsa' ./jiveSBS <to be provided>:/
        rsync instructions no host.png
    4. Ensure the correct paths to both your private key file and to the actual binstore (./jiveSBS in the example above). They will likely differ from the example above.
    5. You may chose to set up a cron job to execute this synchronization periodically after the initial run
    6. You may chose to add the -P parameter to the rsync command after -av to show the progress of the synchronization
    7. Note that incorrect usage of this command may result in files you already uploaded being deleted. If that happens, correct the paths in the command and run rsync again. In particular, don't provide a trailing slash for the topmost local folder you are trying to synchronize, or the contents of that folder will be considered the top level contents on the target RSYNC, wiping out any other folders you may have synchronized, such as database backups from a different folder.

     

    Notes on the graphic above

    [1] The server requires that --delete operations are enabled for rsync. You will get an error if this argument is missing. Please ensure that you don't accidentally point to an empty source folder.

    [2] The host name will be provided later. The actual customer-specifc, isolated path where your data is stored is managed and enforced by the secure server.

    Also note that --progress is optional and --partial is suggested so that interrupted large transfers can be resumed.

     

    For questions about this document, please contact Jive Professional Services.