Upgrading a Cluster with a Separated Database

Important: If you are upgrading from 2020.1 S54 or above, you must apply the Upgrade_s54to20221_Windows.zipUpgrade_s54to20221_Linux.zip file to the server in step 5.

Before upgrading, be sure to read the release notes for the new version. The release notes contain a list of known issues, important compatibility information, supported upgrade paths, and module-specific data backup recommendations.
Apply all the latest updates available for your operating system, especially those that resolve issues with Java.
Make sure that you follow the appropriate upgrade path for the currently installed version of Spectrum that you want to upgrade.
Important: We recommend that you create a backup before upgrading so that if an error occurs during the upgrade you can recover your flows, security settings, and other settings and customizations.
To retain customized settings in the wrapper.conf file located in the SpectrumDirectory\server\bin\wrapperSpectrumDirectory/server/bin/wrapper directory, you will need to compare the contents of wrapper.conf installed during the upgrade with the contents of the backed up copy of the file. Before upgrading make sure you have backed up all models and property files. You can then manually copy customizations that you want to retain after the upgrade into the updated version of the file. This is particularly important for changes to the initial and maximum Java heap sizes. This release requires Java 11, so corresponding updates to JVM properties should be used.

This procedure describes how to upgrade Spectrum Technology Platform when you have separated the configuration database from the server. To upgrade an environment that has a separated configuration database, you first upgrade the configuration database cluster, then the server cluster.

Back up the server.
For instructions for creating a backup, see Creating a Backup Manually in the Spectrum Administration Guide.

Important: We recommend that you create a backup before upgrading, so if an error occurs during the upgrade process you can recover your flows, security settings, and other settings,
Back up module-specific data for any of these modules if you have them installed.
Stop all the nodes in the server cluster, then stop all the nodes in the configuration database cluster.
- Right-click the Spectrum Technology Platform icon in the Windows system tray and select Stop Spectrum.
- Run the DatabaseDirectory/server/bin/server.stop script.
For instructions on stopping a cluster, see Stopping a Cluster.
Check that all Spectrum Java processes have been shut down before you start the upgrade.

Note: If you have Global Sentry installed, stop the Global Sentry database server by opening the Windows Services Manager and stopping the Global Sentry Database Server service.
If you have applied the 2020.1 S54 patch or later, install the Upgrade_s54to20221_Windows.zipUpgrade_s54to20221_Linux.zip file to each node in the database configuration cluster before you continue.
For more information, see Upgrade from 2020.1 S54 patch or later.
On each node in the cluster, run installdb.shinstalldb.exe to upgrade the configuration database.

Note: If the database automatically starts after upgrading a node, stop the configuration database.
After the upgrade is complete, configure each node in the database cluster with the following settings.
1. Edit the file DatabaseDirectory/repository/spectrum-container.propertiesDatabaseDirectory\repository\spectrum-container.properties as described in Cluster Properties for a Configuration Database Cluster.
2. Confirm that the settings in the DatabaseDirectory/repository/neo4j.templateDatabaseDirectory/repository/neo4j.template file match the following settings:
```
dbms.active_database=graphdb
dbms.default_database=graphdb
dbms.default_listen_address=0.0.0.0
causal_clustering.discovery_advertised_address=${spectrum.agent.address}
causal_clustering.enable_pre_voting=false
dbms.routing.enabled=true
dbms.routing.listen_address=0.0.0.0
dbms.routing.advertised_address=${spectrum.agent.address}
```
Perform the following steps to copy the graphdb folder from one node to the other nodes in the cluster:
1. Back up and delete graphdb from these repository folders on node-2, node-3, …
  
  DatabaseDirectory/repository/data/databases
  DatabaseDirectory/repository/data/transactions
  
  DatabaseDirectory\repository\data\databases
  DatabaseDirectory\repository\data\transactions
2. From node-1, copy the repository databases /graphdb from this directory to the other nodes in the cluster (node-2, node-3, …):
  DatabaseDirectory/repository/data/databasesDatabaseDirectory\repository\data\databases
3. From node-1, copy the repository transactions graphdb from this directory to the other nodes in the cluster (node-2, node-3, …):
  DatabaseDirectory/repository/data/transactions DatabaseDirectory\repository\data\transactions
Start each node consecutively in the configuration database cluster.
To start the configuration database, right-click the Spectrum Technology Platform icon in the Windows system tray and select Start Spectrum.

Run the DatabaseDirectory/server/bin/server.start script.

Perform the remaining steps to upgrade the server cluster.

Back up the server.
For instructions on creating a backup, see the Administration Guide.
Check that all Spectrum Java processes on each machine are stopped before you start the upgrade.

Note: If you have Global Sentry installed, stop the Global Sentry database server by opening the Windows Services Manager and stopping the Global Sentry Database Server service.
Upgrade each node in the server cluster.
On each node, perform the following steps.
1. Open the file SpectrumDirectory\server\config\spectrum-container.propertiesSpectrumDirectory/server/config/spectrum-container.properties in a text editor and configure the cluster properties.
  
  For more information, see Cluster Properties.
2. Open the file SpectrumDirectory/index/elasticsearch.templateSpectrumDirectory\index\elasticsearch.template and scroll down to the Node Address section.
  - Insert the hash character # to comment out the following line:
```
# cluster.initial_master_nodes: ${initial_master_node}
```
  - Remove the hash character # and the space after it to uncomment the following line
```
cluster.initial_master_nodes: ${seeds}
```
    Important: Make sure that there is no space at the beginning of the line so that the code starts in column 1.
    
    Then edit it to specify the node names in the cluster:
```
cluster.initial_master_nodes: node-1,node-2,node-3,...
```
    Important: This setting must be uncommented when you start the upgraded cluster for the first time.
3. Confirm the following settings on each node in the cluster.
  - SpectrumDirectory/server/conf/spectrum-container.propertiesSpectrumDirectory\server\conf\spectrum-container.properties
```
spectrum.repository.cluster.mode=CAUSAL
```
    Note: The spectrum.repository.cluster.mode setting shown here is for the Spectrum Technology Platform server running in a cluster.
  - Increase the timeout setting in SpectrumDirectory/server/conf/spectrum-container.propertiesSpectrumDirectory\server\conf\spectrum-container.properties file from 120 to 1200.
```
spectrum.repository.server.startup.timeout=1200
```
Check again that all Spectrum Java processes on each machine are stopped before you start the cluster.
Start each node in the server cluster consecutively.
To start a server, right-click the Spectrum Technology Platform icon on the Windows task bar, and click Start Spectrum.

To start the Spectrum Technology Platform server, run the SpectrumDirectory/server/bin/server.start script.

Do not wait for one node to start before starting another node. All nodes must be started at the same time.

For more information, see Starting a Cluster.
After you have upgraded and started all the nodes, make the following changes on each node.
- Open the file SpectrumDirectory/index/elasticsearch.templateSpectrumDirectory\index\elasticsearch.template, and insert the hash character # to comment out the following line:
```
# cluster.initial_master_nodes: node-1,node-2,node-3,...
```
  Important: Comment out this setting after you start the upgraded cluster for the first time. It should then remain commented out for all subsequent restarts of the upgraded cluster, as Elasticsearch handles adding and removing nodes for subsequent restarts of the upgraded cluster.

Note: Any job or subflow stored in the system during a server upgrade is marked as exposed to provide the same behavior for those jobs and subflows as before the upgrade. Any job or subflow that is exported prior to the upgrade will not be exposed. Therefore, when importing these jobs or subflows back into the system, you must manually expose the imported job or subflow.

The following modules have post-upgrade steps you need to perform.

Module	Post-upgrade
Spatial	During an upgrade, the pool size is reset to 4 if set to a lower value. A pool size setting of 4 or more is unaffected. To change the Remote Component Pool Size value after upgrading, open the Management Console and select Resources > Spatial. Change the pool size value and click Save. For details, see Modifying the Pool Size in the Spectrum Spatial Guide.
Context Graph	See Upgrade Data Hub or Context Graph or Upgrading a Cluster with Context Graph.