- Questions to Answer when Experiencing Performance Issues
- Logs to Provide to Jive Support for Troubleshooting
- Client-side Logs
- Server Logs
- Related Materials
This document outlines the most important piece of information Jive Support will need when customers report performance issues with the Jive application when running Jive in an On Premise enivronment.
If your community is unavailable or experience poor performance please reach out to Jive Support immediately by opening a new Severity 1 support case in your secret MyJive support group. Proactively providing the logs listed below will help expedite the case investigation.
Questions to Answer when Experiencing Performance Issues
If you have a report of a performance issue, please provide answers to the following question to help Jive understand the scope of the issue and the nature of the performance impact. Having this information upfront will help expedite the troubleshooting process and move the investigation one step closer to resolution.
Questions to answer when experiencing performance issues in Jive:
- Is the issue happening for everyone or specific people?
- Is the issue happening on site-wide or specific pages?
- Is the issue happening constantly, or is there a specific time period?
- Is the issue happening on all nodes or specific nodes?
- Note: This request is for On-Prem instances only - Jive can help determine this information for Hosted and Cloud customers
Logs to Provide to Jive Support for Troubleshooting
There are a variety of log files and diagnostics that can be provided to Jive to help aid a support case investigation. Depending on the nature of the problem different logs will be more helpful.
Logs and Diagnostics can either be gathered on the client-side browser, or the Jive server(s) / database server.
The general recommendation is that for most performance related issues, the jive-httpd-access.log file(s) and a thread dump taken during the performance event are absolute requirements for understanding why a system may be suddenly slow. It is critical to gather the thread dump before restarting your Jive application. If these files are not gathered during the performance event then Jive may not be able to provide an RCA.
Types of Logs and Diagnostics
|Log Type||When Is This Needed?|
|Client-side Logs and Diagnostics to Gather|
|Browser Network Requests (.har or fiddler)||General troubleshooting|
|Server Logs and Diagnostics to Gather|
|sbs.log - Jive Application Server Logs||General troubleshooting|
|jive-httpd-access.log - Jive HTTPD Access Logs||General troubleshooting|
|Garbage Collection Logs||JVM / Memory Issues|
|Heap Dump||JVM / Memory Issues|
|Thread Dump||CPU Issues|
|Top Output||CPU Issues|
|Oracle AWR Report (Oracle DB Only)||Oracle Database performance Issues|
|Slow Database Queries (PostgreSQL, MySQL, SQL Server)||Database performance issues|
|sar (System Activity Report)||General troubleshooting|
Browser Network Requests
Nearly all browsers now include developer tools in their default installs that can give you detailed information as to the time it takes for the requests to be rendered by the server (one slice of time) and for requests to be downloaded by your browser (yet another slice of time). Additionally, these network requests include data sent and received by the browser.
Instructions for capturing network requests in Chrome, Safari, and Firefox: How to Capture Web Requests (HAR) in Chrome for... | Jive Community
Instructions for using Fiddler to capture network requests in IE: https://community.jivesoftware.com/docs/DOC-74864
Here's a screenshot of what Chrome Developer Tools (Settings --> Tools --> Developer Tools) looks like when narrowly focusing just on the amount of time it takes to render the homepage of a Jive application:
There are a number of interesting things the can be extrapolated from this tool:
a) in the Timeline column, you can see that there are two parts to the bar: the light blue is the amount of time it took the server to render the homepage and is a strict measure of how quickly the application is generating the page. The darker blue is the amount of time it took the browser to download the HTML source for the generated page and is more a measure of network latency between the browser (wherever the host is located) and the server (which may be located many timezones away from the browser). If the light blue dominates the response time, then the problem is most likely server related. If however, the dark blue dominates this view, then the problem is most likely some kind of network latency, which most likely will need to be solved by someone in your IT department that has visibility into the various networks (local, remote, internet) that live between the browser and the Jive server(s).
b) if you click on the page (i.e. /Welcome) in the above screenshot, you'll be presented with an extensive list of information about the HTTP transaction including the HTTP request and response headers as well as any information included via HTTP POST, here's an example page view:
This shows what page was requested, where the user was previously (the Referer value), what cookies, if any, where sent along with the request, what cookies, if any, were sent back by the server, any custom HTTP header sent as part of the request or as part of the response, as well as the HTTP Status code and HTTP Request Method. All of this information is usually extremely helpful when attempting to debug low level HTTP performance issues.
The sbs.log file is helpful to see any errors logged by the Jive application. If there are errors being captured in these logs then the support team may be able to use these to determine how to address the issue. The errors may also suggest that the performance issues are being caused by another resource, like low disk space, database connectivity, etc.
The jive-httpd-access.log file is the apache http server log, which contains a list of all web requests coming into the Jive application.
This file is required for almost all performance related issues and is necessary to complete an RCA.
Garbage Collection Logs
Every node in the Jive infrastructure that's running Java (i.e. the web node(s), the cache server(s), the activity engine server(s), the document conversion server(s), and the search server(s)) includes a log file that records each run of the Java garbage collector which makes it easy to spot, at a glance, if one of the services is spending a lot of time doing garbage collection, which is then the most likely cause of a performance slow down. The garbage collection log file will be located in the following directory:
and there may be multiple garbage collection log files. You can locate the current garbage collection log file by executing the following command from a shell prompt:
> ps -ef | grep Jive
which will then print out something like this:
jive 11909 1 83 Sep08 ? 1-09:30:21 JiveApplication -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+DisableExplicitGC -XX:NewRatio=2 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Xloggc:/usr/local/jive/var/logs/exampleco-gc-2013-09-08_183403.log -Djive.text.extract.dir=/mnt/jive_binstore/jiveSearchText -Djava.security.egd=file:///dev/urandom -Xmx8192m -Xms8192m -XX:MaxPermSize=328m -Djive.home=/usr/local/jive -Djive.instance.home=/usr/local/jive/applications/exampleco/home -Djive.name=exampleco -Djive.context=/ -Djive.logs=/usr/local/jive/var/logs -Djive.application=/usr/local/jive/applications/exampleco/application -Djive.work=/usr/local/jive/var/work/exampleco -Dserver.port=9100 -Dhttp.addr=0.0.0.0 -Dhttp.port=9200 -Dhttp.monitor.addr=0.0.0.0 -Dhttp.monitor.port=9000 -Dlog4j.configuration=file:///usr/local/jive/applications/exampleco/conf/log4j.properties -Dapp.cluster.jvmroute= -Dcom.sun.management.jmxremote.port=6650 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djive.text.extract.dir=/mnt/jive_binstore/jiveSearchText -Dcatalina.base=/usr/local/jive/applications/exampleco -Dcatalina.home=/usr/local/jive/tomcat -Djava.io.tmpdir=/usr/local/jive/var/work/exampleco -classpath /usr/local/jive/applications/exampleco/bin//bootstrap.jar:/usr/local/jive/applications/exampleco/bin/tomcat-juli.jar: org.apache.catalina.startup.Bootstrap start
The line above in bold includes the full path to the current garbage collection log file, which is a simple text file. There are a number of free tools that you can use to analyze the log file but simply gathering that log file and including it in the set of log files that you provide to Jive support will be sufficient for analyzing heap issues. If however, you'd like to analyze the garbage collection trends yourself, we highly recommend a tool from HP called HPjmeter, which is available here:
If the garbage collection log shows that the node is spending a lot of time doing garbage collection, which will usually manifest itself as either a Full GC or a Concurrent Mark Sweep in the GC logs, then you'll most likely be requested to gather a heap dump, which is a bit more involved than simply looking at the log file. Heap dumps are useful when attempting to identify the cause of OutOfMemoryErrors and/or poor performance. Heap dumps can be quite laborious to capture, transfer, and analyze, so Support will not usually ask for them unless there is a real need.
Jive recommends using the jmap tool, freely available from Oracle, for gathering the heap dump:
Note that the heap dump, if the invocation of jmap is successful, will result in a large file, which is usually about the same size as the amount of heap that you've configured the Java process with. The Jive support team can provide you with information on how to use FTP to upload the resulting heap dump to a secured customer file upload space.
Finally, if you reboot the server or servers that are experiencing garbage collection issues BEFORE gathering a heap dump, the resulting heap dump will not be useful and it's likely that performing a thorough root cause analysis of the issue will be much more difficult, if not impossible since the details as to what was consuming memory on heap has been lost as the result of the restart of the Java process.
Thread dumps are a "snapshot" of all active Java processing threads in the Java Virtual Machine. Thread dumps are often the only tool which can help an engineer identify certain concurrency problems such as deadlocks. They also can be quite useful in determining the nature of the problem of a "hung" application.
Tools for the Job:
- Capture in Jive 7 or newer:
- Example command: jive snap -c 5 -i 5
- Documentation: Jive 7.0 Community Admin Documentation
- Capturing in Jive 6 or older
- Example command: appsnap -c 5 -i 5
- Documentation: Jive 6.0 Community Administrator Documentation
- Tool for thread dump analysis:
- TDA (for analysis)
Thread dumps taken during the event is required for almost all performance related issues and is necessary to complete an RCA.
Usually the Top output is included by the jive snap tool mentioned above, but if you choose not to use jive snap and / or if for some reason jive snap does not include it, it's helpful to gather Linux top output, which will show if there are other processes running on the nodes in question that are consuming CPU. This is unlikely but has proven to be useful in some cases.
Oracle AWR Report
If you are running an Oracle database on-premise and the case investigation suggests that there is an issue with database performance or connectivity, then the Jive Support team will request that your Oracle DBA produces an Oracle AWR (Automatic Workload Repository) report. Details on how to create and access the AWC report are available here:
The generated report is usually an HTML file and should be uploaded to the Jive support case.
Slow Database Queries
If you are running your own MySQL, Postgres, or SQL Server then it is highly recommended that you configure your database server to log queries that take longer than 500ms to a slow query log.
Having a list of slow database queries can help the support team identify if there is a part of the Jive database architecture or application is needs to be tuned for your community.
sar (System Activity Report)
sar is a Linux tool which gathers critical system data and stores it in files in /var/log/sa/ (may vary by distribution). This tool offers a convenient way to view a snapshot of critical system metrics over a given time period. Visualize sar data with ksar or another tool to view a profile of your system's performance and identify trouble areas in your hardware/software configuration.