How to Configure Cross Realm Trust for HBase

Background

If two Hadoop clusters with Kerberos applied have different REALMs, data transfer between the two clusters is theoretically impossible. However, by configuring Cross Realm Trust, it becomes possible to query data across clusters or freely move data between them.

Prerequisite

We will assume that there are two clusters, A and B, with Kerberos already applied. Cluster A belongs to the UBUNTU.YJ.COM realm, and Cluster B belongs to the CENTOS.YJ.COM realm. Each cluster has Zookeeper, HDFS, HBase, and YARN components.

Steps

1. Creating krbtgt

In cross realm setups, the principal named krbtgt is extremely important.

Register the following two principals in the UBUNTU.YJ.COM KDC:

krbtgt/UBUNTU.YJ.COM@CENTOS.YJ.COM krbtgt/CENTOS.YJ.COM@UBUNTU.YJ.COM

Similarly, register the following two principals in the CENTOS.YJ.COM KDC:

krbtgt/UBUNTU.YJ.COM@CENTOS.YJ.COM krbtgt/CENTOS.YJ.COM@UBUNTU.YJ.COM

The encryption type, password, and kvno of all four principals must be identical.

2. Updating /etc/krb5.conf and /etc/hosts

On servers belonging to the UBUNTU.YJ.COM realm, you need to add the CENTOS.YJ.COM related configuration to /etc/krb5.conf as follows.

[libdefaults]
  default_realm = UBUNTU.YJ.COM
[realms]
  UBUNTU.YJ.COM = {
    kdc = kdc01.hadoop.example.com:88
    admin_server = kdc01.hadoop.example.com:749
    default_domain = ubuntu.yj.com
  }
  CENTOS.YJ.COM = {
    kdc = kdc01.example.com:88
    admin_server = kdc01.example.com:749
    default_domain = centos.yj.com
  }
[domain_realm]
  .ubuntu.yj.com = UBUNTU.YJ.COM
  ubuntu.yj.com = UBUNTU.YJ.COM
  .centos.yj.com = CENTOS.YJ.COM
  centos.yj.com = CENTOS.YJ.COM

On servers belonging to the CENTOS.YJ.COM realm, you need to add the UBUNTU.YJ.COM related configuration to /etc/krb5.conf as follows.

[libdefaults]
  default_realm = CENTOS.YJ.COM
[realms]
  CENTOS.YJ.COM = {
    kdc = kdc01.example.com:88
    admin_server = kdc01.example.com:749
    default_domain = centos.yj.com
  }
  UBUNTU.YJ.COM = {
    kdc = kdc01.hadoop.example.com:88
    admin_server = kdc01.hadoop.example.com:749
    default_domain = ubuntu.yj.com
  }

[domain_realm]
  .centos.yj.com = CENTOS.YJ.COM
  centos.yj.com = CENTOS.YJ.COM
  .ubuntu.yj.com = UBUNTU.YJ.COM
  ubuntu.yj.com = UBUNTU.YJ.COM

Also, add the information about the other cluster's servers to /etc/hosts.

3. kvno test

Authenticate with a keytab on Cluster A and test whether you can access Cluster B's server using kvno.

kinit -kt <A keytab> <A principal>
kvno <B principal>

Authenticate with a keytab on Cluster B and test whether you can access Cluster A's server using kvno.

If the kvno value is displayed as an integer properly, the cross realm configuration has been completed successfully.

4. Adding auth to local Rules

When /etc/krb5.conf is changed, all related components (HBase, HDFS, YARN) must be restarted. At this time, the following config must be added to core-site.xml on Cluster A.

<property>
    <name>hadoop.security.auth_to_local</name>
    <value>RULE: [1:$1@$0](.*@\QCENTOS.YJ.COM\E$)s/@\QCENTOS.YJ.COM\E$//
    RULE: [2:$1@$0](.*@\QCENTOS.YJ.COM\E$)s/@\QCENTOS.YJ.COM$//
    DEFAULT</value>
</property>

Add the following to Cluster B.

<property>
    <name>hadoop.security.auth_to_local</name>
    <value>RULE: [1:$1@$0](.*@\QUBUNTU.YJ.COM\E$)s/@\QUBUNTU.YJ.COM\E$//
    RULE: [2:$1@$0](.*@\QUBUNTU.YJ.COM\E$)s/@\QUBUNTU.YJ.COM$//
    DEFAULT</value>
</property>

For understanding the auth_to_local syntax, refer to this page.

5. test

Once the restart is complete, you can proceed with migration tasks such as export snapshot, replication setup, and verify replication from Cluster A to Cluster B or from Cluster B to Cluster A.

Reference

• https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_sg_kdc_def_domain_s2.html
• https://www.youngju.dev/blog/202211/KDC_server_install
• https://community.cloudera.com/t5/Community-Articles/Auth-to-local-Rules-Syntax/ta-p/245316

Quiz