Deep-dive into Multi-cluster with Linkerd, Terraform, and Naver Cloud.
Imagine you’re the Head of Engineering at a fast-growing startup. Your product, The Pegasus Project, is going viral. The user base is surging across EMEA and the US, and now the company is expanding operations to South Korea. With hundreds of thousands of users already onboard, you’re now preparing to support thousands of new users in South Korea.
The first step in this expansion will be an assessment where you will evaluate the implications of scaling globally, considering various trade-offs, challenges, and risks. This expansion won’t happen overnight, so before moving forward, you’ll likely present a plan to your stakeholders.
The Assessment
In this assessment, you’ll most likely take a long list of factors into account. In this article, we will limit them to the following:
Latency and Region-Lock
With new users in South Korea, latency becomes crucial. The easiest way to reduce latency is to run services as close as possible to the end-users, which means, in South Korean data centers.
Currently, managed Kubernetes services like Amazon EKS, Google GKE, and Azure AKS, and their node pools are tight to a regions where they can enjoy the high availability of the availability zones in the region. While this is not affecting private infrastructure as you have more flexibility, it will introduce more challenges like networking and security.
Additionally, even if one day (which is unlikely due to their hierarchical structures like region, availability zone, data center) Cloud Service Providers decide to support multi-region node pools, I’d still recommend against binding your architecture to a single vendor, as it will make you dependent on their policies and availability. By remaining vendor-agnostic, you could take advantage of local cloud providers discounts and leverage stronger regional support, and reduce potential disruptions if you decide to switch to another CSP.
Compliance and Regulation
Expanding globally means dealing with data sovereignty and regional regulations; in this case South Koreans and the CSAP (Cloud Security Assurance Program) certification.
The CSAP was introduced in 2016, and it’s managed by the Korean Internet & Security Agency (KISA). It cover data protection policies for public sector cloud services, incident management, and physical security, and its mandatory to being able to participate in the projects related to government agencies and other public bodies, such as schools and hospitals. One of the painful points for CSPs like Azure, AWS, and GCP is the strict physical separation of public and private cloud servers for government agencies.
Even if in January 2024 the Korean government has been pushing for greater private cloud adoption and started to ease the requirements to get this certification, and splitting it into three tiers: high, medium, and low. If the Pegasus Project primary focus will be government entities, local providers like Naver Cloud, NHN Cloud, and Kakao Cloud would be mandatory choices.
Complexity and Scaling Challenges
As your cluster expands, so does its complexity. Managing configurations, policies, and networking across hundreds or thousands of nodes becomes increasingly challenging. Even a routine operation, like performing a rolling API upgrade, can quickly become a complex and extremely long tasks. Additionaly, immagine how complex will be when something will go wrong (becasuse sooner or later something won’t go as expected) having to troubleshoot a cluster with thousands of nodes.
Kubernetes Limitations
Kuberentes comes with best-practice limitations regarding the number of nodes and pods each cluster can handle. For instance, as of Kubernetes version 1.31, the recommended limit is 5,000 nodes per cluster, with a maximum of 110 pods per node. While these are not hard restrictions, and many CSPs like GKE support configurations well beyond this limit, for example GKE can handle up to 15,000 nodes with 256 pods per node, these limits are there for good reason and should be carefully considered. CSP’s SLAs might be working as the service will be running, but Kubernetes API server, etcd, and other components may become slower, and the entire cluster more difficult to troubleshoot.
Multi-Cluster
After weighing these challenges, you decide it’s time to explore a different approach, and this is where multi-cluster architectures come into play. As the name suggests, multi-cluster involves multiple Kubernetes clusters working together as one cohesive system. However, this solution comes with trade-offs that need to be carefully considered:
- Flexibility and Complexity: Multi-cluster setups allow for clusters optimized for specific workloads. However, with varying configurations, engineers will need a deeper understanding to manage different cluster types, each with unique tuning and operational demands.
- Global Reach and Networking Complexity: Moving beyond the CSP regional limits introduces additional networking complexity. You’ll need to enable cross-cluster service discovery and manage resource allocation across clusters to specify where different workloads run. Service meshes like Linkerd can help address this challenge by providing service discovery across clusters.
- Maintainability: Managing multiple clusters increases the risk of configuration drift, as each cluster has its own API, controllers, and operational policies.
- Isolation and improved security: This benefit is particularly effective in a Cooperative Multicluster architecture, where, instead of duplicating the entire application across clusters, we leverage cluster boundaries like such as namespaces by deploying one workload (or a set of related workloads) per cluster. This results in enhanced separation, as any unauthorized RBAC changes made by a malicious user in one cluster, such as cross-namespace access, won’t affect other clusters.
Multi-Cluster and Cluster-Federation
The concept of multi-cluster is not new in the Kubernetes world. Back in 2018, Kubernetes Federation (kubefed) introduced the idea of federated clusters. In a federated approach, a single Kubernetes control plane manages multiple clusters, enabling centralized deployment and workload management across clusters. With Kubernetes Federation, there is effectively a single control plane that coordinates multiple clusters, simplifying operations but also requiring an additional layer of control. However, this approach has been abandoned, and there are no active developments on it.
Multi-Cluster Architectures
In a multi-cluster architecture, each cluster operates independently, with its own control plane and worker nodes. A central load balancer or service mesh connects these clusters, directing traffic based on factors like resource availability or latency requirements. Here are some common multi-cluster architectures you might consider:
- Replicated Architecture with Passive Failover: In this approach, identical copies of the application and services are deployed across clusters. If one cluster becomes unavailable, a secondary cluster takes over, ensuring service continuity.
- Replicated Architecture with Active Failover: Similar to passive failover, but here, the secondary cluster actively processes traffic and requests, providing a load-sharing approach and at the same time providing high availability.
- Split-by-Service (Cooperative Multicluster): Each cluster is designated for specific services. For example, latency-sensitive workloads can be placed in clusters geographically closer to users, while compute-intensive batch jobs run in clusters with greater compute resources, such as GPUs.
Naver Kubernetes Service (NKS)
Ncloud Kubernetes Service (NKS) is Naver Cloud Platform’s managed Kubernetes solution. Similar to AWS EKS, Google GKE, and Azure AKS, NKS minimizes the complexities of cluster setup, operation, and maintenance. It offers built-in functionality for cluster autoscaling, monitoring, and integration with other Naver Cloud services.
Note: At the time of this writing, Naver Cloud still supports NKS in Classic environments but no longer allows the creation of new clusters in this setup.
Node Pools
In NKS, virtual machines with similar configurations are grouped into node pools, with at least one node pool required per cluster. Node pools can be assigned labels and taints, which are used to manage the scheduling of the nodes.
Prerequisites
To create an NKS cluster, there are several resources that must be in place to establish the necessary infrastructure for networking, load balancing, and internet access.
- Private Network: A Virtual Private Cloud (VPC) with a CIDR range between /17 and /26 is required, providing private IP addresses for internal communications within the cluster.
- Operational Subnet: This subnet is dedicated to the core operations of the cluster.
- Internal Load Balancer Subnet: A subnet for the internal load balancer that handles traffic distribution within the cluster using private IP addresses within the VPC. This subnet is inaccessible from outside the VPC, using approximately 5–10 IPs per load balancer.
- External Load Balancer Subnet: This subnet is for the external load balancer, which distributes incoming traffic from outside the cluster. It similarly requires 5–10 IPs per load balancer to manage inbound requests.
- NAT Gateway (Optional): If your cluster needs outbound internet access, a NAT Gateway is necessary so that it can enables nodes to connect to external servers while preventing unsolicited inbound connections, improving security by allowing only authorized external traffic.
Buoyant Enterprise for Linkerd
Linkerd comes in two flavors: Open-Source and Enterprise. The Enterprise edition offers premium features like HAZL, FIPS compliance, and — crucially for many organizations — dedicated support. In high-stakes environments, relying solely on community forums isn’t an option. Having a team of engineers ready to jump on a call during a Severity 1 issue can be invaluable. To meet these needs, Buoyant (the company behind Linkerd) introduced Buoyant Enterprise for Linkerd.
The core architecture is still Linkerd, so if you’re interested in learning more about how Linkerd works, check out my previous article.
Linkerd and Multi-cluster architecture
Linkerd multi-cluster functionality is based on the service mirror Kubernetes Operator which mirrors a remote cluster’s services locally, and provide service discovery so that the pods can refer to the remote services. It also manages configuring endpoints so that traffic goes to the correct IP address.
Since the clusters will use the public internet to talk to each other the communication between them will use the default mTLS encryption provided by linkerd so that the communication is secure. To do so, all clusters will need to share a common trust anchor. The trust anchor certificate is used to issue the issuer certificate that will be used to issue unique certificates to ech one of its proxies. While you can use the same issuer certtificate for all clsuter, it’s a best practice to use a separate issuer certificat for each cluster.
How does service mirroring works?
When you enable the multicluster by installing the multicluster Helm Chart, it will add the CRDs, Roles, and ClusterRoles needed for sevice discovery and synchronization, like list, get, and watch endpoints, endpointSlices, pods, and more. Optionally, it will deploy a Gateway with Service Type Load Balancer that is going to be used by the remote services to access the local pods.
With the these resources in place, you can move forward and link one or more clusters. Upon the execution of the link command in the Linkerd CLI, it will collect the cluster configuration which contains the server location as well as the CA bundle, then fetch the remote linkerd-service-mirror-remote-access-default-token and merge these pieces of configuration into a kubeconfig secret in the local cluster cluster-credentials-REMOTE-CLUSTER-NAME secret (each cluster will have its own secret).
$ kubectl get secrets -A
NAMESPACE NAME TYPE DATA AGE
linkerd-multicluster cluster-credentials-remote mirror.linkerd.io/remote-kubeconfig 1 13m
linkerd-multicluster linkerd-service-mirror-remote-access-default-token kubernetes.io/service-account-token 3 62m
...
$ kubectl get secrets -n linkerd-multicluster cluster-credentials-remote -o yaml
apiVersion: v1
data:
kubeconfig: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
kind: Secret
metadata:
name: cluster-credentials-remote
namespace: linkerd-multicluster
type: mirror.linkerd.io/remote-kubeconfigThis secret is Base64 encoded. To view the information about the remote cluster, such as the server URL, CA data, and authentication token, we can decode it with the following command:
$ kubectl get secret cluster-credentials-remote -n linkerd-multicluster -o jsonpath='{.data.kubeconfig}' | base64 --decode -
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
server: https://aks-training-euw-dns-cefntn63.hcp.westeurope.azmk8s.io:443
name: aks-training-euw
contexts:
- context:
cluster: aks-training-euw
user: linkerd-service-mirror-remote-access-default
name: aks-training-euw
current-context: aks-training-euw
kind: Config
preferences: {}
users:
- name: linkerd-service-mirror-remote-access-default
user:
token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxThe overall flow will be as follows:
Then, it will create a deployment of the service mirror controller, which will use a dedicated service account to access this secret, enabling communication with the Kubernetes API service in the remote cluster.
kubectl get deploy -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
...
linkerd-multicluster linkerd-service-mirror-원격-클러스터 1/1 1 1 10mThis controller will then be in charge of using the remote Kubernetes API Service to collect the definition of the services marked to be mirrored and then create these services in the local cluster. When a service is provisioned, to avoid local naming collisions, Linkerd appends the cluster name to service name of type ClusterIP in the remote cluster name. For example, if we have a service called svc in the local cluster, it will appear in the remote cluster name provided in the link as svc-REMOTE-CLUSTER-NAME.default.svc.cluster.local .
Note: The links are unidirectional. However, you can create multiple links to make it bidirectional.
Both remote and local services are independent from each others and have their own state. However, the ClusterIP service in the local cluster does not directly connect to the pods in the remote cluster, and even if the IPs from the remote clusters were copied to the local endpoint resource, they won’t be reachable, as they don’t exist in the local cluster. To solve this issue, Linkerd provides two different networking modes: Hierarchical and Flat.
Hierarchical networks
This is the default mode used by linkerd if not specified otherwise. In fact, the default values of the Helm Chart will automatically deploy and configurure the Gateway as LoadBalancer or NodePort service, which will expose it to the internet.
gateway:
enabled: true
name: linkerd-gateway
port: 4143
serviceType: LoadBalancer # NodePort
...In hierarchical, the endpoint resource in the local cluster won’t have the IPs of the pods in the remote cluster, but just the IP of the gateway deployed in the remote cluster. All requests in the local cluster will then be encrypted using mTLS and sent to the remote gateway. When the remote gateway receives these requests, it verifies their authenticity and source, allowing only verified requests through.
Going though the gateway will add an additional network hop which will increase the latency. Additionally, because the requests are coming from the gateway and not the source workload the authorization will be challenging.
Flat networks
To address issues related to latency and authorization, Linkerd introduced pod-to-pod connectivity with the flat network in version 2.14. In this mode, requests go directly from pod to pod without passing through the gateway. However, to enable this direct communication, clusters need distinct IP address ranges (CIDR) to ensure connectivity between arbitrary pods when linking them. Although the communication is direct, the connection remains secure because all data is encrypted with mTLS.
Another advantage is that, because requests come directly from a workload, the server can apply authorization policies to allow only specific clients to connect.
Terraform
Naver Cloud is expanding its range of services rapidly, and the Terraform provider is continuously updated to support more of these offerings. While not all resources are currently available in the provider, the ones we’ll be using in this tutorial are fully supported.
To initialize the Naver Cloud provider in Terraform, you’ll need to create an access key and a secret key. Navigate to your account settings, select Manage Auth Key, and click Create a New API Authentication Key. Once you have these credentials ready, you can proceed with configuring Terraform.
In this tutorial, I’ll use modules from the terraform-navercloudplatform-modules repository. These modules, which I developed to enhance code reusability and add validations, will streamline the process of provisioning resources. If you’re interested, I’ve included a link to the repository in the reference section at the end of this article.
terraform {
required_providers {
ncloud = {
source = "NaverCloudPlatform/ncloud"
}
}
required_version = ">= 0.13"
}
provider "ncloud" {
access_key = var.access_key
secret_key = var.secret_key
region = var.region
support_vpc = true
}
data "ncloud_nks_server_images" "image" {
hypervisor_code = "KVM"
filter {
name = "label"
values = ["ubuntu-22.04"]
regex = true
}
}
data "ncloud_nks_server_products" "product" {
software_code = data.ncloud_nks_server_images.image.images[0].value
zone = "KR-1"
filter {
name = "product_type"
values = ["STAND"]
}
filter {
name = "cpu_count"
values = ["2"]
}
filter {
name = "memory_size"
values = ["8GB"]
}
}
module "vpc" {
source = "terraform-navercloudplatform-modules/vpc-vpc/ncloud"
version = "1.0.0"
name = "example-vpc"
ipv4_cidr_block = "10.0.0.0/16"
}
module "subnet" {
source = "terraform-navercloudplatform-modules/subnet-vpc/ncloud"
version = "1.0.1"
name = "example-subnet"
vpc_no = module.vpc.id
subnet = "10.0.1.0/24"
zone = "KR-1"
subnet_type = "PRIVATE"
usage_type = "GEN"
network_acl_no = module.vpc.default_network_acl_no
}
module "subnet_lb" {
source = "terraform-navercloudplatform-modules/subnet-vpc/ncloud"
version = "1.0.1"
name = "example-subnet"
vpc_no = module.vpc.id
subnet = "10.0.100.0/24"
zone = "KR-1"
subnet_type = "PRIVATE"
usage_type = "LOADB"
network_acl_no = module.vpc.default_network_acl_no
}
module "login_key" {
source = "terraform-navercloudplatform-modules/login-key/ncloud"
version = "1.0.1"
key_name = "example-key"
}
module "cluster" {
source = "terraform-navercloudplatform-modules/kubernetes-cluster-vpc/ncloud"
version = "v1.0.1"
name = "example-cluster"
vpc_no = module.vpc.id
subnet_no_list = [module.subnet.id]
lb_private_subnet_no = module.subnet_lb.id
cluster_type = "SVR.VNKS.STAND.C002.M008.NET.SSD.B050.G002"
login_key_name = module.login_key.name
zone = "KR-1"
}
module "node_pool" {
source = "terraform-navercloudplatform-modules/kubernetes-node-pool-vpc/ncloud"
version = "1.0.0"
cluster_uuid = module.cluster.uuid
node_pool_name = "example-node-pool"
node_count = 2
software_code = data.ncloud_nks_server_images.image.images[0].value
server_spec_code = data.ncloud_nks_server_products.product.products.0.value
storage_size = 200
autoscale = {
enabled = false
min = 2
max = 2
}
}For the second cluster, we will deploy a new cluster in Azure using the Azure Kubernetes Service in West Europe with the following Terraform configuration.
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = ">= 2.0.0"
}
}
required_version = ">= 0.13"
}
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "resource_group" {
name = "rg-training-krc"
location = "West Europe"
}
resource "azurerm_kubernetes_cluster" "kubernetes_cluster" {
name = "aks-training-euw"
location = azurerm_resource_group.resource_group.location
resource_group_name = azurerm_resource_group.resource_group.name
dns_prefix = "trainingaks"
default_node_pool {
name = "default"
node_count = 1
vm_size = "Standard_D2_v2"
}
identity {
type = "SystemAssigned"
}
}Next, execute terraform apply and provision the resources. As you can see even if we are targeting different CSPs, the cloud agnostic nature of Terraform, makes it an amazing tool to streamline the provisioning process of your infrastructure.
Installing Linkerd Enterprise
With the infrastructure in place, we can proceed with installing the Linkerd service mesh in both clusters. In this article, we will use Helm charts, but you can find more installation options in my previous article:
First, we will add the Helm repository and install the Linkerd Custom Resource Definitions (CRDs) Helm chart:
helm repo add linkerd-buoyant https://helm.buoyant.cloud
helm repo update
helm upgrade --install linkerd-enterprise-crds linkerd-buoyant/linkerd-enterprise-crds \
--namespace linkerd \
--create-namespaceTo enable mTLS, Linkerd requires several certificates to be configured before installing the control plane, specifically a trust anchor and an issuer certificate. The following commands will generate these certificates using step:
step certificate create root.linkerd.cluster.local ./certificates/ca.crt ./certificates/ca.key \
--profile root-ca \
--no-password \
--insecure
step certificate create identity.linkerd.cluster.local ./certificates/issuer.crt ./certificates/issuer.key \
--ca ./certificates/ca.crt \
--ca-key ./certificates/ca.key \
--profile intermediate-ca \
--not-after 8760h \
--no-password \
--insecure Finally, we can install the control plane, specifying the previously created certificates and the enterprise license:
helm upgrade --install linkerd-control-plane linkerd-buoyant/linkerd-enterprise-control-plane \
--version 2.16.2 \
--set-file linkerd-control-plane.identityTrustAnchorsPEM=./certificates/ca.crt \
--set-file linkerd-control-plane.identity.issuer.tls.crtPEM=./certificates/issuer.crt \
--set-file linkerd-control-plane.identity.issuer.tls.keyPEM=./certificates/issuer.key \
--set license=$BUOYANT_LICENSE \
--namespace linkerd \
--create-namespace Once completed, you should see the following pods running in your clusters:
kubectl get pods -n linkerd
NAME READY STATUS RESTARTS AGE
linkerd-destination-9c78b68d6-mxn7v 4/4 Running 0 13m
linkerd-enterprise-7d698fd4d-57xxn 2/2 Running 0 13m
linkerd-identity-56df89dbf9-66hcq 2/2 Running 0 13m
linkerd-proxy-injector-85b59cb786-p56tj 2/2 Running 0 13mInstall and Link the cluster via Linkerd Multicluster using Hierarchical networks mode and LoadBalancer
Now it’s time to connect our clusters. To do so, we’ll need to install the Linkerd multicluster Helm chart. In this tutorial, we’ll use the default gateway mode.
Note: This chart is available in both OSS and Enterprise versions. The Enterprise version is a wrapper around the Linkerd OSS chart, allowing you to configure it using the values available in the OSS chart.
helm install linkerd-enterprise-multicluster linkerd-buoyant/linkerd-enterprise-multicluster \
--namespace linkerd-multicluster \
--create-namespaceThe chart will install new CRDs (links.multicluster.linkerd.io and AuthorizationPolicy), disallow any traffic not originating from a Linkerd-meshed identity, installs the Gateway deployment and necessary roles.
$ kubectl get pods -n linkerd-multicluster
NAME READY STATUS RESTARTS AGE
pod/linkerd-gateway-5cc499df-sxk29 2/2 Running 0 22m
$ kubectl get service -n linkerd-multicluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/linkerd-gateway LoadBalancer 10.0.101.238 4.158.39.42 4143:31386/TCP,4191:32565/TCP 27m
$ kubectl get secrets -n linkerd-multicluster
NAMESPACE NAME TYPE DATA AGE
default linkerd-service-mirror-remote-access-default-token kubernetes.io/service-account-token 3 28m
$ kubectl get sa -n linkerd-multicluster
NAME SECRETS AGE
linkerd-gateway 0 5m38s
linkerd-service-mirror-remote-access-default 0 5m38sAdditionally, it will create secret with a token and certificate authority which will be used when linking this cluster to another one.
$ kubectl get secret linkerd-service-mirror-remote-access-default-token -o yaml
apiVersion: v1
data:
ca.crt: xxxxxxxxxxxxxxx
namespace: xxxxxxxxxxxxxxx
token: xxxxxxxxxxxxxxx
kind: Secret
metadata:
...
name: linkerd-service-mirror-remote-access-default-token
type: kubernetes.io/service-account-tokenOnce the installation is complete, we can proceed to link our clusters. The CLI is handy here, as it deploys all necessary resources for the link.
Note: The context used in the link command is for the remote cluster, while kubectl apply uses the local context, giving the local cluster access to remote services.
kubectl config use-context remote
linkerd multicluster link --cluster-name="remote" > remote-link.yaml
kubectl config use-context local
kubectl apply -f remote-link.yamlIn the local cluster, you’ll see several new resources, including a ClusterIP service with the remote gateway’s public IP as an endpoint, a service mirror pod, and secrets specific to this link.
$ kubectl get svc -n linkerd-multicluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
linkerd-gateway LoadBalancer 10.0.233.104 85.210.41.71 4143:32479/TCP,4191:31164/TCP 11m
probe-gateway-remote ClusterIP 10.0.140.234 <none> 4191/TCP 10m
$ kubectl get endpoints -n linkerd-multicluster
NAME ENDPOINTS AGE
linkerd-gateway 10.244.1.35:4191,10.244.1.35:4143 11m
probe-gateway-remote 85.210.73.136:4191 10m
$ kubectl get pods -n linkerd-multicluster -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
linkerd-gateway-8b6b7f5df-68n8m 2/2 Running 0 7m6s 10.244.1.35 agentpool-75896186 <none> <none>
linkerd-local-service-mirror-c6fbf4b57-9vjlh 2/2 Running 0 7m6s 10.244.1.85 agentpool-75896186 <none> <none>
linkerd-service-mirror-remote-7d98d467cd-pz86w 2/2 Running 0 6m8s 10.244.1.56 agentpool-75896186 <none> <none>
$ kubectl get secrets -n linkerd-multicluster
NAME TYPE DATA AGE
cluster-credentials-remote mirror.linkerd.io/remote-kubeconfig 1 11m
linkerd-service-mirror-remote-access-default-token kubernetes.io/service-account-token 3 12m
sh.helm.release.v1.multicluster.v1 helm.sh/release.v1 1 12m
$ kubectl get link.multicluster.linkerd.io -n linkerd-multicluster
NAME AGE
remote 13mInspecting the link resource reveals details about the identity, remote gateway public IP, ports, and other required information for connectivity.
$ kubectl get link.multicluster.linkerd.io -n linkerd-multicluster remote -o yaml
apiVersion: multicluster.linkerd.io/v1alpha2
kind: Link
metadata:
name: remote
namespace: linkerd-multicluster
...
spec:
clusterCredentialsSecret: cluster-credentials-remote
gatewayAddress: 85.210.73.136
gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
gatewayPort: "4143"
probeSpec:
failureThreshold: "3"
path: /ready
period: 3s
port: "4191"
timeout: 30s
remoteDiscoverySelector:
matchLabels:
mirror.linkerd.io/exported: remote-discovery
selector:
matchLabels:
mirror.linkerd.io/exported: "true"
targetClusterDomain: cluster.local
targetClusterLinkerdNamespace: linkerd
targetClusterName: remoteAdditionally, if we check the service mirror created from the link, we will see that the proxy will take care of adding endpoint to the remote gateway linkerd proxy. If something will go wrong with the connectivity, this is the service that we will need to check.
$ kubectl logs -n linkerd-multicluster linkerd-service-mirror-remote-7d98d467cd-pz86w
[ 20569.078745s] INFO ThreadId(01) outbound:proxy{addr=85.210.73.136:4191}:service{ns=linkerd-multicluster name=probe-gateway-remote port=4191}: linkerd_pool_p2c: Adding endpoint addr=85.210.73.136:4191Next, we’ll deploy and mesh an application in the remote cluster, which will later be mirrored in the local cluster.
After its deployment, even though the application is meshed, we won’t see any reference to it in the local cluster. This is because, for mirroring to be triggered, specific labels need to be applied to the service depending on the mirroring mode. In this case, we are using hierarchical networks mode, so we need to add the following label to the service:
mirror.linkerd.io/exported:trueOnce the label is added, the mirrored service will appear in the local cluster.
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
vastaya application-vastaya-svc-remote ClusterIP 10.0.171.130 <none> 80/TCP 23mChecking the endpoint confirms that it targets the public IP of the gateway in the remote cluster.
$ kubectl get endpoints -A
NAMESPACE NAME ENDPOINTS AGE
...
vastaya application-vastaya-svc-remote 85.210.73.136:4143 24m
$ kubectl get endpoints -n vastaya application-vastaya-svc-remote -o yaml
apiVersion: v1
kind: Endpoints
metadata:
annotations:
mirror.linkerd.io/remote-gateway-identity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
mirror.linkerd.io/remote-svc-fq-name: application-vastaya-svc.vastaya.svc.cluster.local
labels:
mirror.linkerd.io/cluster-name: remote
mirror.linkerd.io/mirrored-service: "true"
name: application-vastaya-svc-remote
namespace: vastaya
subsets:
- addresses:
- ip: 85.210.73.136
ports:
- name: http
port: 4143
protocol: TCPFinally, if we send a request to the mirrored service, we’ll receive a response from the remote pods. This confirms that the request reaches the gateway in the remote cluster. Since the gateway is meshed with the Linkerd proxy, the usual workflow applies, with the proxy intercepting both incoming and outgoing traffic and, in this case, forwarding the request to the target pod.
kubectl exec -it -c ubuntu ubuntu -- curl "http://application-vastaya-svc-remote.vastaya.svc.cluster.local"
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Web site created using create-react-app"/><link rel="apple-touch-icon" href="/logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Training</title><script defer="defer" src="/static/js/main.084249d4.js"></script><link href="/static/css/main.792f89a7.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>Install and Link the cluster via Linkerd Multicluster using Hierarchical networks mode and NodePort
In some environments, you might face limitations where a LoadBalancer service type isn’t an option, which can pose challenges for cross-cluster communication. Since the multicluster gateway essentially acts as an ingress, bridging isolated networks to enable communication even when they lack direct connectivity Linkerd Multicluster accommodates this scenario by allowing different types of services, like NodePort, to function as the gateway for cross-network connectivity.
First, we will need to set several values inthe multicluster helm chart so that it will create the gateway as NodePort with defined ports and probes ports.
helm upgrade --install multicluster linkerd/linkerd-multicluster \
--set gateway.serviceType=NodePort \
--set gateway.nodePort=32000 \
--set gateway.probe.nodePort=32001 \
--create-namespace \
--namespace linkerd-multiclusterOn some cloud providers, like Azure, nodes don’t automatically receive public IPs, which will prevent the communication cross-clusters. To make this connection working we will need to enable public IPs during cluster creation or if you have already created a cluster, then create a new nodepool with the “Enable public IP per node” option enable.
Additionally, by default, the network security groups ibound roles will allow just traffic from the vnet. We will need to add a new rule allowing inboud traffic from the port used by the nodePort.
Next, during the link creation, we will need to specify the IP of the node/s as well as their ports.
$ kubectl config use-context remote
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
nodepool-32050982 Ready <none> 21m v1.29.10 10.224.0.6 20.254.89.100 Ubuntu 22.04.5 LTS 5.15.0-1074-azure containerd://1.7.23-1
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
linkerd-multicluster linkerd-gateway NodePort 10.0.151.213 <none> 4143:32000/TCP,4191:32001/TCP 44m
$ linkerd multicluster link --cluster-name="remote" --gateway-addresses 20.254.89.100 --gateway-port 32000 > remote-link.yamlProceed by applying the link resources in the local cluster, and add the mirror.linkerd.io/exported=true label to the remote service you want to mirror. When the mirrored service appears in the local cluster, check the endpoints to confirm that it points to the configured NodePort service.
$ kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
vastaya application-vastaya-svc-remote ClusterIP 10.0.33.125 <none> 80/TCP 11s
$ kubectl get endpoints -A
NAMESPACE NAME ENDPOINTS AGE
...
linkerd-multicluster probe-gateway-remote 20.254.89.100:32001 18m
vastaya application-vastaya-svc-remote 20.254.89.100:32000 14mOnce traffic is generated from the local cluster, you will receive a response from the remote pods.
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vastaya projects-vastaya-svc-remote ClusterIP 10.0.233.220 <none> 80/TCP 17h
$ kubectl exec -it -c ubuntu ubuntu -- curl http://projects-vastaya-svc-remote.vastaya.svc.cluster.local
[{"id":1,"name":"Mock Project","description":"This is a mock project for demonstration purposes","status":"open"}]This response confirms a successful connection because the request originates from a Linkerd-meshed pod sharing the same trust anchor. A request from outside the mesh (e.g., your local machine) would be refused due to a lack of mTLS encryption:
$ curl 20.254.89.100:32000
curl: (52) Empty reply from serverInspecting the remote gateway logs will reveal that the connection was denied due to unauthenticated access.
[ 32521.541274s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_inbound::policy::http: Request denied server.group= server.kind=default server.name=all-authenticated route.group= route.kind=default route.name=default client.tls=None(NoClientHello) client.ip=10.224.0.6
[ 32521.543808s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:rescue{client.addr=10.224.0.6:62647}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=unauthorized request on route
[ 40215.702987s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_inbound::policy::http: Request denied server.group= server.kind=default server.name=all-authenticated route.group= route.kind=default route.name=default client.tls=None(NoClientHello) client.ip=10.224.0.6
[ 40215.703039s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:rescue{client.addr=10.224.0.6:59273}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=unauthorized request on route
[ 50077.970355s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.224.0.6:31935 server.addr=10.244.2.154:4143Installing and Linking Clusters with Linkerd Multicluster Using Flat Network Mode
Flat network mode doesn’t require a gateway. To skip gateway deployment, configure the multicluster Helm chart as follows:
helm upgrade --install multicluster linkerd/linkerd-multicluster \
--set gateway.enabled=false \
--create-namespace \
--namespace linkerd-multiclusterBy default, the link command will check for the existance of the gateway. For this reason, we will need to instruct it to skip this check.
Once completed, we can then add the annotation to the remote service that we want to be exportd to the local cluster. In this case it will be:
mirror.linkerd.io/exported=remote-discoveryFinally, if we generate some traffic, in the local traffic, we be redirected by the linkerd proxy directly to the remote pods without passing through the gateway.
References:
- Kuberenets limitations: https://kubernetes.io/docs/setup/best-practices/cluster-large/
- Azure AKS limitations: https://learn.microsoft.com/en-us/azure/aks/quotas-skus-regions
- Kubefed: https://github.com/kubernetes-retired/kubefed
- Linkerd Multicluster: https://linkerd.io/2-edge/tasks/installing-multicluster/
- Naver Cloud Terraform Modules: https://registry.terraform.io/namespaces/terraform-navercloudplatform-modules
