Inspecting a Non-Responsive Server

Spectrum runtime consists of multiple JVM processes. The primary process is the Spectrum server process. When the primary server process is not responding, Spectrum does not respond to requests. However, each remote component runs in a separate process. When a remote component process stops responding, only the corresponding functions do not work correctly. Other processes may continue to work. The following steps identify the process to check when there is no response to a request.

Identify which Process is not Responding

Spectrum Management Console: Connect to the Spectrum Management Console (http://<server>:<port>/managementconsole). If the Spectrum Management Console loads and functions as expected, then the main Spectrum server process is OK.

Spatial Services: On the Spectrum Technology Platform software page (http://<server>:<port>/dcg), select Spectrum Spatial from the menu. If the Spectrum Spatial service demo page loads and works as expected, then the Spectrum Spatial remote component process is OK.

http://<servername>:<port>/Spatial/MappingService/DemoPage.html
http://<servername>:<port>/Spatial/FeatureService/DemoPage.html

WebDAV API: Connect to the Repository WebDAV API (http://<server>:<port>/RepositoryService/repository/default) to ensure the repository is working as expected.

Enable JMX

After identifying the process that may have a problem, enable JMX monitoring on that process.

For the Spectrum server process, add the following system properties in %Spectrum%/server/bin/wrapper/wrapper.conf and restart the Spectrum server. For a Spectrum Spatial remote component, add the following system properties in the Spectrum Management Console by selecting from the main menu Resources and Spatial, and then selecting Process arguments.

-Djava.rmi.server.hostname=<hostname>
-Dcom.sun.management.jmxremote.port=9990
-Dcom.sun.management.jmxremote.rmi.port=9990
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Connect to the JMX Server

Launch JConsole (%java%\bin\jconsole.exe ) from any workstation that has a compatible Java SDK installed on it, and that can access Spectrum directly on the network. In JConsole, select Remote process and use the connection string, <server>:9990. If an SSL (secure sockets layer) warning displays, select insecure connection.

Inspect the Process

Monitoring JVM memory usage
When Java Virtual Machine (JVM) is running out of memory (OOM), the process is blocked by garbage collection (GC). Check the JVM memory usage on the Memory tab in JConsole. If the memory size causes the server to be non-responsive, refer to the instructions under Increasing Heap Memory for the Spatial Remote Component to increase the heap size (memory).
Make a heap dump of the JVM process
A memory leak may be one of the reasons that cause OOM. A JVM heap dump (file) can help identify memory leak issues. To find the process ID and generate a heap dump (file), log on to the server machine where Spectrum is running.
```
ps aux | grep java | grep " -Dspatial" (or " -Dspectrum"  for the primary server process)
jmap -dump:format=b,file=<file-path> <pid>
```
For example: %java%/bin/jmap -dump:format=b,file=./heapdump.bin 12345
Make a thread dump of the JVM process
Thread deadlock may be one of the reasons that prevent the process from responding. A JVM thread dump (file) can help identify any deadlocks. Log on to the server machine where Spectrum is running to find the process ID and generate a thread dump (file).
```
ps aux | grep java | grep " -Dspatial"
jstack -F <pid> > <file-path>
```
For example: %java%/bin/jstack -F 12345 > ./threaddump.txt