Upgrading a Cluster

Important: If you are upgrading from 2020.1 S54 or above, you must apply the Upgrade_s54to20221_Windows.zipUpgrade_s54to20221_Linux.zip file to the server in step 5.

Before upgrading, be sure to read the release notes for the new version. The release notes contain a list of known issues, important compatibility information, supported upgrade paths, and module-specific data backup recommendations.
Apply all the latest updates available for your operating system, especially those that resolve issues with Java.
Make sure that you follow the appropriate upgrade path for the currently installed version of Spectrum that you want to upgrade.
Important: We recommend that you create a backup before upgrading so that if an error occurs during the upgrade you can recover your flows, security settings, and other settings and customizations.
To retain customized settings in the wrapper.conf file located in the SpectrumDirectory\server\bin\wrapperSpectrumDirectory/server/bin/wrapper directory, you will need to compare the contents of wrapper.conf installed during the upgrade with the contents of the backed up copy of the file. Before upgrading make sure you have backed up all models and property files. You can then manually copy customizations that you want to retain after the upgrade into the updated version of the file. This is particularly important for changes to the initial and maximum Java heap sizes. This release requires Java 11, so corresponding updates to JVM properties should be used.

This procedure is for upgrading a cluster where the Spectrum Technology Platform server and configuration database are installed on each node of the cluster. To upgrade a cluster, you upgrade one node at a time. The first node you upgrade is handled slightly differently than the other nodes because you must point the node to itself as a seed node since no other nodes will be running in the cluster when it starts up.

The following scenarios have special procedures for upgrading a cluster.

For this scenario	Use this information
Separate clusters for server nodes and configuration database nodes	Upgrading a Cluster with a Separated Database.
Upgrading a cluster for Spatial only	Upgrading a Cluster with Spatial
Upgrading both Spectrum and Spatial clusters	Upgrading a Cluster with Spatial
Upgrading a cluster running Context Graph	Before shutting down all nodes in the cluster, see Upgrading a Cluster with Context Graph.

If none of the scenarios apply to your installation of Spectrum, complete the following steps to upgrade your cluster.

Back up the server.
For instructions on creating a backup, see the Administration Guide.
Back up module-specific data for any of these modules if you have them installed.
Stop all the nodes in the cluster.
For more information, see Stopping a Cluster.
Check that all Spectrum Java processes on each machine are stopped before you start the upgrade.

Note: If you have Global Sentry installed, stop the Global Sentry database server by opening the Windows Services Manager and stopping the Global Sentry Database Server service.
If you have applied the 2020.1 S54 patch or later, install the Upgrade_s54to20221_Windows.zipUpgrade_s54to20221_Linux.zip file on each node before you upgrade Spectrum.
For more information, see Upgrade from 2020.1 S54 patch or later.
Upgrade each node in the cluster.
For more information, see Spectrum Server Upgrade.
On each node, perform the following steps.
1. Open the file SpectrumDirectory/server/conf/spectrum-container.propertiesSpectrumDirectory\server\conf\spectrum-container.properties in a text editor and configure the cluster properties.
  
  For more information, see Cluster Properties.
  Note: Be aware that the container property definitions are dependent upon your server configuration and whether you are running clusters on Neo4j instances. Review the spectrum.repository.server.cluster.nodeCount property to determine the definitions for your setup.
2. Open the file SpectrumDirectory/index/elasticsearch.templateSpectrumDirectory\index\elasticsearch.template and scroll down to the Node Address section.
  - Insert the hash character # to comment out the following line:
```
# cluster.initial_master_nodes: ${initial_master_node}
```
  - Remove the hash character # and the space after it to uncomment the following line
```
cluster.initial_master_nodes: ${seeds}
```
    Important: Make sure that there is no space at the beginning of the line so that the code starts in column 1.
    
    Then edit it to specify the node names in the cluster:
```
cluster.initial_master_nodes: node-1,node-2,node-3,...
```
    Important: This setting must be uncommented when you start the upgraded cluster for the first time.
3. Confirm the following settings on each node in the cluster.
  - SpectrumDirectory/repository/neo4j.template
```
dbms.active_database=graphdb
dbms.default_database=graphdb
dbms.default_listen_address=0.0.0.0 causal_clustering.discovery_advertised_address=${spectrum.agent.address}
causal_clustering.enable_pre_voting=false
dbms.routing.enabled=true
dbms.routing.listen_address=0.0.0.0 dbms.routing.advertised_address=${spectrum.agent.address}
```
  - SpectrumDirectory/server/conf/spectrum-container.properties
```
spectrum.repository.cluster.mode=CAUSAL
```
    Note: The spectrum.repository.cluster.mode setting shown here is for the Spectrum Technology Platform server running in a cluster.
  - Increase the timeout setting in SpectrumDirectory/server/conf/spectrum-container.properties file from 120 to 1200.
```
spectrum.repository.server.startup.timeout=1200
```
Perform the following steps to copy the graphdb from one node to the other nodes in the cluster.
1. Back up and delete graphdb on the other nodes (node-2, node-3, …) from the following repository folders:
  SpectrumDirectory/repository/data/databasesSpectrumDirectory\repository\data\databases
  
  SpectrumDirectory/repository/data/transactionsSpectrumDirectory\repository\data\transactions
2. From node-1, copy the repository databases graphdb to the other nodes in the cluster (node-2, node-3, …) in the following folder:
  SpectrumDirectory/repository/data/databasesSpectrumDirectory\repository\data\databases
3. From node-1, copy the repository transactions graphdb to the other nodes in the cluster (node-2, node-3, …) in the following folder:
  SpectrumDirectory/repository/data/transactionsSpectrumDirectory\repository\data\transactions
Check again that all Spectrum Java processes on each machine are stopped before you start the cluster.
Start each node in the cluster consecutively.
To start a server, right-click the Spectrum Technology Platform icon on the Windows task bar, and click Start Spectrum. Do not wait for one node to start before starting another node. All nodes must be started at the same time.

For more information, see Starting a Cluster.
After you have upgraded and started all the nodes, make the following changes on each node.
- Open the file SpectrumDirectory/index/elasticsearch.templateSpectrumDirectory\index\elasticsearch.template, and insert the hash character # to comment out the following line:
```
# cluster.initial_master_nodes: node-1,node-2,node-3,...
```
  Important: Comment out this setting after you start the upgraded cluster for the first time. It should then remain commented out for all subsequent restarts of the upgraded cluster, as Elasticsearch handles adding and removing nodes for subsequent restarts of the upgraded cluster.