Configuring HDFS Connection for HA Cluster

To configure HDFS connection to HA cluster, specify these details:
  1. Host: <nameservice of cluster>
  2. Port: <Port on which the nameservice is running>.
    1. This value is a combination of the defined dfs.http.policy and the port used for cluster configuration. For example, if dfs.http.policy is HTTPS, the HTTPS port used in cluster configuration needs to be defined here. However, if dfs.http.policy is HTTP, the HTTP port used during cluster configuration will be entered here.
    2. The port value in the following Advanced server options will be same as that defined here.
      • dfs.namenode.http-address.<nameservice name>.<namenode1 name>
      • dfs.namenode.http-address.<nameservice name>.<namenode2 name>
      • dfs.namenode.https-address.<nameservice name>.<namenode1 name>
      • dfs.namenode.https-address.<nameservice name>.<namenode2 name>
  3. Set these key-value pairs in Advanced server options.
    Note: It assumes HTTPS port 9871 has been used for cluster configuration.
    Key Value
    fs.defaultFS hdfs://<nameservice-name>
    dfs.nameservices <nameservice-name>
    dfs.ha.namenodes<nameservice-name> master1, master2 (names of two master nodes)
    dfs.namenode.http-address.<nameservice name>.<namenode1 name> <master1 IP or name>: 9871
    dfs.namenode.http-address.<nameservice name>.<namenode2 name> <master2 IP or name>: 9871
    dfs.namenode.rpc-address.<nameservice name>.<namenode1 name> <master1 IP or name>: 8020
    dfs.namenode.rpc-address.<nameservice name>.<namenode2 name> <master2 IP or name>: 8020
    dfs.client.failover.proxy.provider org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
    dfs.namenode.servicerpc-address.<nameservice name>.<namenode1 name> <master1 IP or name>: 8022
    dfs.namenode.servicerpc-address.<nameservice name>.<namenode2 name> <master2 IP or name>: 8022
    dfs.namenode.https-address.<nameservice name>.<namenode1 name> <master1 IP or name>: 9871
    dfs.namenode.https-address.<nameservice name>.<namenode2 name> <master2 IP or name>: 9871
    hadoop.security.authentication Kerberos
    dfs.namenode.kerberos.principal <principal> for example, master1@HADOOP.COM
    dfs.http.policy HTTPS
    dfs.ha.automatic-failover.enabled true
  4. Make this entry in the Wrapper.conf file at: SpectrumDirectory/server/bin/wrapper: wrapper.java.additional.<n>=-Djavax.net.ssl.trustStore=<truststore file location>/<name of the truststore file>
    Note: Here, n can be any available location and SpectrumDirectory is the directory where you have installed the Spectrum Technology Platform server.