8 Replies Latest reply on May 15, 2013 5:37 AM by miloskrocian

    Filesystem Storage, Archiving, Versioning

    bobjewell

      We're facing an issue with database size, and it brought to mind some possible feature enhancements that have been discussed previously, and I'd appreciate some clarification/updates/suggestions. 

       

      • Currently all file attachments (other than, optionally, image attachments) are stored in blobs in the database.   I'd like to know if there is any progress or intent to optionally use file-storage.   Database space is quite a bit more expensive than filesystem space.. If I have to move my database off to a NAS, then all my database queries are hitting the network, while the majority of the queried info can be contained within 50m or even a gig, while my attachments take up many gigs of blob space.

       

      • I also understand that there is no compression of these attachments.  Office2007 files are already compressed, but a routine process that compresses old versions of files could save space without a performance impact.

       

      • I recently read a post about limiting number of versions stored.  This could be a valuable space saver, as old versions of many documents are unimportant.  We have a great example here of a spreadsheet that is updated several times per day over the past couple weeks.  We're already up to 70 versions.  I can see my users doing something like this routinely for months or years, and ending up with thousands of versions.  I'd like the ability to set a default 'number of versions to keep', and then users could, when viewing 'manage versions', select larger numbers of versions to keep from a dropdown box.  An additional nicety would be to allow users to flag version to store indefinitely, from that same screen.

       

      • Is there a mechanism to archive old "spaces" (projects?)  out of the database?   My back-of-napkin brainstorming was to have an archive process that resulted in dumping a compressed single file "image" of a space into an archival location.   In my case, this would conveniently be an NFS mount that has a more relaxed backup schedule.   In the future, if I wanted to access one of those spaces, I could browse the 'archived spaces' table in the admin console, and re-invoke that space somewhere in my space tree.  It would be re-invoked 'in archive' mode, where everything is locked, and read-only, with clear labeling that is it an archive of a certain space from this certain date.    My only option now is to delete the space.. losing the content.

       

        • Re: Filesystem Storage, Archiving, Versioning

          Thanks for the questions Bob. Responses below.

           

          • Currently all file attachments (other than, optionally, image attachments) are stored in blobs in the database.   I'd like to know if there is any progress or intent to optionally use file-storage.   Database space is quite a bit more expensive than filesystem space.. If I have to move my database off to a NAS, then all my database queries are hitting the network, while the majority of the queried info can be contained within 50m or even a gig, while my attachments take up many gigs of blob space.

           

          At the moment file attachments are still stored in the database. We have several customers bringing up this point with us, so it is on the radar, but we don't have a specific plan of action yet.

           

          • I also understand that there is no compression of these attachments.  Office2007 files are already compressed, but a routine process that compresses old versions of files could save space without a performance impact.

           

          Great idea, we'll see what we can do to add some compression into the mix.

           

          • I recently read a post about limiting number of versions stored.  This could be a valuable space saver, as old versions of many documents are unimportant.  We have a great example here of a spreadsheet that is updated several times per day over the past couple weeks.  We're already up to 70 versions.  I can see my users doing something like this routinely for months or years, and ending up with thousands of versions.  I'd like the ability to set a default 'number of versions to keep', and then users could, when viewing 'manage versions', select larger numbers of versions to keep from a dropdown box.  An additional nicety would be to allow users to flag version to store indefinitely, from that same screen.

           

          Thanks for the vote on this one, it definitely seems like a good idea.

           

          • Is there a mechanism to archive old "spaces" (projects?)  out of the database?   My back-of-napkin brainstorming was to have an archive process that resulted in dumping a compressed single file "image" of a space into an archival location.   In my case, this would conveniently be an NFS mount that has a more relaxed backup schedule.   In the future, if I wanted to access one of those spaces, I could browse the 'archived spaces' table in the admin console, and re-invoke that space somewhere in my space tree.  It would be re-invoked 'in archive' mode, where everything is locked, and read-only, with clear labeling that is it an archive of a certain space from this certain date.    My only option now is to delete the space.. losing the content.

           

          Not yet, but it is something we've been considering, especially for projects. At the moment we've been thinking about it as archiving but leaving in the database, but I can see how the idea of archiving into a file- and directory-based bundle would be beneficial.

           

          Thanks again,

          Greg

            • Re: Filesystem Storage, Archiving, Versioning
              pcosta

              Hello there. I'd like to vote for the filesystem-based attachments too. It should be done in a way that it would benefit incremental backups too.

              Is there a formal way to place this vote?

              Thanks!

              Pablo

                • Re: Filesystem Storage, Archiving, Versioning

                  Thanks Pablo, this is great input. By participating here you're effectively voting for this kind of change.

                    • Re: Filesystem Storage, Archiving, Versioning
                      bradwarren

                      I'd like to chime in on this one too and place votes for both filesystem storage of attachments and especially for the archiving of spaces and projects.  At my company, we're a service firm that is project based and we're using Clearspce subspaces (and now Projects) for each project.  We've been using Clearspace for a year and things are already starting to get cluttered.  I currently "archive" subspaces by moving them under a space named "Archive" that only admins can see.  At least that way most of our users don't get flooded with project subspaces.  But the Projects feature doesn't have the ability to move, hide or archive in any way.  We would love to be able to do that!  And even better would be the ability to archive a space or project all the way to a file and remove it from the database.

                       

                      Thanks!

                        • Re: Filesystem Storage, Archiving, Versioning

                          Hi Brad,

                           

                          Thanks for chiming in! We're in the midst of detailed release planning and had a great conversation yesterday about moving projects around. It's looking like that will make it into the next release. The ability to archive projects, spaces and groups won't be, but is in our longer-term plans.

                           

                          Best Regards,

                          Greg

                    • Re: Filesystem Storage, Archiving, Versioning
                      sivo_kb

                      I'd like to vote for the file-storage option as well. We'd like to share some huge (multi-GB) files among users, which just isn't feasible in a database BLOB.

                      -Bruce

                    • Re: Filesystem Storage, Archiving, Versioning

                      Hi I'd like to "vote" or see the file storage option for files. I think allot of admins would like to at the very least have a choice in how files are stored. Some people may have large files or a large number of files and being able to store them outside the database can bring many advantages from costs to flexibility.

                       

                      Cheers,

                       

                      Stephen.

                       

                      PS. loving the 2.5 release!

                      • Re: Filesystem Storage, Archiving, Versioning
                        miloskrocian

                        Some updates on this? I started similar discussion Some tips for storing large ammount of data/files in Jive? as we are facing similar problems right now.