4 Replies Latest reply on Feb 25, 2008 11:17 PM by ajohnson1200

    Database migration utility fails - Java heap out of memory

      We're trying to stabilize a clearspace production server that has a long upgrade history (since 1.4) that is exhibiting numerous instabilities and errors.

       

      The first goal I have is to migrate the server to a new stable host configuration:

      (CentOS 5.1, linux 2.6.18-8.1.8.el5,  mysql 5.0.22, jdk 1.6u4).

      The current config is identical except is running jdk 1.6u3.

      When I try to run the migration utility in the admin screens, it crashes with the following exception:

       

      • Status Code: 500

      • Exception Type:

      • Error Message:

      • Request URI: /clearspace/admin/database-migration-task.jspa

      • Stack Trace:

        • sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        • sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        • sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        • java.lang.reflect.Method.invoke(Method.java:597)

        • com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:358)

        • com.opensymphony.xwork.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:218)

        • com.opensymphony.webwork.interceptor.BackgroundProcess$1.run(BackgroundProcess.java:28)

        • java.lang.Thread.run(Thread.java:619)

      • java.lang.reflect.InvocationTargetException

       

      Here's a snip of the last valid screen that is displayed before the exception occurs:

       

       

       

      Here's the Datasource Info from the existing installation:

       

      Database Name and Version:

      MySQL

                     5.0.22

      JDBC Driver and Version:

      MySQL-AB JDBC Driver mysql-connector-java-5.0.7 ( $Date: 2007-03-09

                     22:13:57 +0100 (Fri, 09 Mar 2007) $, $Revision: 6341 $ )

      Subqueries supported:

      Yes

      Transactions supported:

      No

      Connection Pool:

      Default Clearspace Connection Pool

      Min DB Connections:

      5

      Max DB Connections:

      50

      DB Connections Open:

      5

      Active DB Connections:

      2

       

      Here's the relevant log snippet related to the incident:

       

      Feb 21, 2008 12:25:29 PM com.opensymphony.webwork.dispatcher.DispatcherUtils serviceAction

      SEVERE: Could not execute action

      java.lang.reflect.InvocationTargetException

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:358)

      at com.opensymphony.xwork.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:218)

      at com.opensymphony.webwork.interceptor.BackgroundProcess$1.run(BackgroundProcess.java:28)

      at java.lang.Thread.run(Thread.java:619)

      Caused by: java.lang.OutOfMemoryError: Java heap space

      at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2457)

      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)

      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:885)

      at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1360)

      at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2369)

      at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:451)

      at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2076)

      at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1451)

      at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1787)

      at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)

      at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313)

      at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1448)

      at com.jivesoftware.community.importer.ResultSetIterator.getBlock(ResultSetIterator.java:150)

      at com.jivesoftware.community.importer.ResultSetIterator.fillCache(ResultSetIterator.java:108)

      at com.jivesoftware.community.importer.ResultSetIterator.hasNext(ResultSetIterator.java:69)

      at com.jivesoftware.community.migration.MigrateTable.performStep(MigrateTable.java:130)

      at com.jivesoftware.community.migration.DatabaseMigrationRoutine.start(DatabaseMigrationRoutine.java:54)

      at com.jivesoftware.community.action.admin.DatabaseMigrationTaskAction.execute(DatabaseMigrationTaskAction.java:48)

       

      Our servlet container is the one bundled with CS 1.10.1 standalone, and is configured in start-clearspace.sh as follows:

       

      JAVA_OPTS="-Xms512m -Xmx1024m -XX:MaxPermSize=256m -XX:UseParNewGC -XX:UseConcMarkSweepGC -server -Djava.awt.headless=true -DjiveHome=$/jiveHome"

       

      Currently I'm guessing it's either a host configuration problem, or possibly an error in the metadata.

      Any light can be shed on how to fix the problem would be much appreciated!

       

      Message was edited by: ccranford (cleaned up pasted HTML mangled by RTE)

        • Re: Database migration utility fails - Java heap out of memory

          hi Caleb,

           

          Key line in the stack trace:

          Caused by: java.lang.OutOfMemoryError: Java heap space

          The database migration tool doesn't read everything from your database at once, it does it's best to scroll through the results, but I think your best bet would be to increase the heap you allocate to the JVM while you're going through the migration (you probably can decrease it to the settings that you have after the migration is complete).

           

          Cheers,

           

          AJ

          1 person found this helpful
            • Re: Database migration utility fails - Java heap out of memory

              AJ, thanks for the response.

               

              I was aware that the problem had do to with the JVM's heap, as indicated by the thread title.

               

               

               

              I should have mentioned that I have already tried allocating 2GB to the JVM as follows:

               

               

               

              -Xms1024m -Xmx2048m

               

              This is the max amount available to user processes on the host.

              Do you have any metrics on the average amount of memory required to run the migration script?

              • Re: Database migration utility fails - Java heap out of memory

                I believe I have a solution, though it may be a hack. I was going through the logs and observed this message on startup:

                 

                WARNING: Java system property java.awt.headless has not been set to true. This will cause graphics related features such as avatars to fail.

                 

                 

                 

                This seemed to be in contrast with the settings I mention above for JAVA_OPTS.

                Looking closer at start-clearspace.sh, I notice that JAVA_HOME and CATALINA_PID are exported env variables, but JAVA_OPTS was not.

                 

                 

                 

                When I changed JAVA_OPTS="..." to export JAVA_OPTS="..." and restarted, the headless warning goes message goes away, the memory parameters are loaded correctly, and I am able to run the migration utility. Great!

                 

                 

                 

                As I mentioned, I recently inherited the maintenence of this service, and I have no frame of reference for how things should be, and what might or might not have been hacked in the past.

                 

                 

                 

                Can you clarify if this a bug in the start-clearspace.sh script, or just a flaw local to our installation?