1 Reply Latest reply on Nov 30, 2016 12:07 PM by jesse.fuller

    Activity engine stalling

    jacques

      We are using Jive 8.0.0 and we found our ectivity engine stalling.

       

      The EAE service seems up but no new events are being shown in the GUI.

       

      We do see some error messages.

       

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:50)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

      2016-11-28 10:18:53,857 [pool-8-thread-2] ERROR [e8dcbebf-75e2-4dbd-8723-e735e782d767] com.jivesoftware.service.phasedqueue.TwoPhasedFileQueueImpl - we don't have permissions to write to var/queues/activityQueue/dcf1a330-b50b-11e6-bfee-00155d93c9ac

      2016-11-28 10:18:53,857 [pool-8-thread-2] ERROR [e8dcbebf-75e2-4dbd-8723-e735e782d767] com.jivesoftware.eae.service.queue.QueueImpl - Failed to serialize: Activity{user=User{userId=2206}, systemID='e8dcbebf-75e2-4dbd-8723-e735e782d767', time=1480324733842, type='viewed', context=EAEEntityDescriptor{id=2134, type=14}, descriptor=EAEEntityDescriptor{id=3565, type=102}, parent=EAEEntityDescriptor{id=3565, type=102}, slug='', content='', activityID=ActivityID{uuid=[-82, -94, 55, 32, -75, 75, 17, -26, -87, 16, 0, 21, 93, -109, -55, -84]}, properties={suppressedUnreadUsers=[], taggedParent=[EAEEntityDescriptor{id=0, type=0}], tagEntities=[], taggedContent=[EAEEntityDescriptor{id=0, type=0}], source=[EAEEntityDescriptor{id=0, type=0}], contributors=[], authors=[EAEEntityDescriptor{id=2197, type=3}]}, additionalProperties={}, forUser=null, localeCode='en', streamFamilies=null}

      java.io.IOException: invalid permissions

              at com.jivesoftware.service.phasedqueue.TwoPhasedFileQueueImpl.append(TwoPhasedFileQueueImpl.java:216)

              at com.jivesoftware.service.phasedqueue.FileQueueImpl.add(FileQueueImpl.java:149)

              at com.jivesoftware.eae.service.queue.QueueImpl.enqueue(QueueImpl.java:133)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.add(ActivityManagerImpl.java:289)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.addAll(ActivityManagerImpl.java:181)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:154)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:146)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:56)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:50)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

      2016-11-28 10:18:53,857 [pool-8-thread-2] ERROR [e8dcbebf-75e2-4dbd-8723-e735e782d767] com.jivesoftware.eae.service.impl.ActivityManagerImpl - Failed to write activity to queue, rolling back firehose

      2016-11-28 10:18:53,860 [pool-8-thread-2] ERROR [e8dcbebf-75e2-4dbd-8723-e735e782d767] com.jivesoftware.eae.service.impl.ActivityManagerImpl - Error while adding activity, it must be resent later.

      com.jivesoftware.eae.service.queue.FailedEnqueueException: java.io.IOException: invalid permissions

              at com.jivesoftware.eae.service.queue.QueueImpl.enqueue(QueueImpl.java:144)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.add(ActivityManagerImpl.java:289)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.addAll(ActivityManagerImpl.java:181)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:154)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:146)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:56)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:50)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

      Caused by: java.io.IOException: invalid permissions

              at com.jivesoftware.service.phasedqueue.TwoPhasedFileQueueImpl.append(TwoPhasedFileQueueImpl.java:216)

              at com.jivesoftware.service.phasedqueue.FileQueueImpl.add(FileQueueImpl.java:149)

              at com.jivesoftware.eae.service.queue.QueueImpl.enqueue(QueueImpl.java:133)

              ... 10 more

      2016-11-28 10:18:53,860 [pool-8-thread-2] ERROR [e8dcbebf-75e2-4dbd-8723-e735e782d767] com.jivesoftware.eae.service.impl.ActivityManagerImpl - Exception processing bulk activity addition.

      com.jivesoftware.eae.service.queue.FailedEnqueueException: java.io.IOException: invalid permissions

              at com.jivesoftware.eae.service.queue.QueueImpl.enqueue(QueueImpl.java:144)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.add(ActivityManagerImpl.java:289)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.addAll(ActivityManagerImpl.java:181)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:154)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:146)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:56)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:50)

              at java.util.concurrent.FutureTask.run(FutureTask.java:266)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

      Caused by: java.io.IOException: invalid permissions

              at com.jivesoftware.service.phasedqueue.TwoPhasedFileQueueImpl.append(TwoPhasedFileQueueImpl.java:216)

              at com.jivesoftware.service.phasedqueue.FileQueueImpl.add(FileQueueImpl.java:149)

              at com.jivesoftware.eae.service.queue.QueueImpl.enqueue(QueueImpl.java:133)

              ... 10 more

      2016-11-28 10:18:53,860 [New I/O  worker #2] ERROR [e8dcbebf-75e2-4dbd-8723-e735e782d767] com.jivesoftware.eae.sendreceive.EAEServerHandler - Service request failed

      com.jivesoftware.eae.service.queue.FailedEnqueueException: java.io.IOException: invalid permissions

              at com.jivesoftware.eae.service.queue.QueueImpl.enqueue(QueueImpl.java:144)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.add(ActivityManagerImpl.java:289)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl.addAll(ActivityManagerImpl.java:181)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:154)

              at com.jivesoftware.eae.service.impl.ActivityManagerImpl$2.apply(ActivityManagerImpl.java:146)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:56)

              at com.jivesoftware.eae.sendreceive.command.TimeoutSendCommand$1.call(TimeoutSendCommand.java:50)

       

      Restarting the EAE service eliviates the problem.