Add explanation for Pod IP addresses.
This commit is contained in:
Tom Alexander
parent
fbb8376ccc
commit
f3c22c18e5
GPG Key ID:
D3A179C9A53C0EDE
96
README.md
96
README.md
@ -11,6 +11,8 @@ REF "GKE networking model doesn't allow IP addresses to be reused across the net
|
||||
|
||||
REF "Combining multiple Ingress resources into a single Google Cloud load balancer is not supported." : https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
|
||||
|
||||
REF "At minimum, the nonMasqueradeCIDRs property should include the node and Pod IP address ranges of your cluster." : https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent
|
||||
|
||||
TOOD: replace tf with terraform
|
||||
|
||||
GKE IP Address Usage Demo
|
||||
@ -21,9 +23,9 @@ This repo contains a terraform configuration that demonstrates efficient use of
|
||||
TL;DR
|
||||
-----
|
||||
- Service IP addresses are not accessible outside a cluster (TODO REF)
|
||||
- Pod IP addresses by default are not accessible outside a cluster (TODO REF)
|
||||
- Pod IP addresses can be contained to the cluster by configuring SNAT to the node IP addresses.
|
||||
- Therefore, we can use (TODO ranges) for pods and services (TODO REF)
|
||||
- This is recommended by Google (TODO REF)
|
||||
- [This is recommended by Google](https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-kubernetes-pod-ip-allocation-in-gke)
|
||||
|
||||
What is spun up
|
||||
---------------
|
||||
@ -159,12 +161,12 @@ Explanation
|
||||
To conserve the RFC-1918 address space, we need to take advantage of two facts:
|
||||
|
||||
1. Service IP addresses aren't real
|
||||
2. Pod IP addresses do not need to leave the cluster (and by default they do not on GKE)
|
||||
2. Pod IP addresses do not need to leave the cluster
|
||||
|
||||
Service IP Addresses
|
||||
--------------------
|
||||
|
||||
Service IP addresses are a fiction created by kubernetes. Service IP addresses are not routable from outside the cluster and packets to service IP addresses are never written to the wire. When a pod sends a packet to a service IP address, it is intercepted by iptables which perform DNAT to the pod's IP address. We can see this on our GKE cluster by connecting to the compute engine instance for a node over `ssh` and inspecting its iptables rules.
|
||||
Service IP addresses are a fiction created by kubernetes. Service IP addresses are not routable from outside the cluster and packets to service IP addresses are never written to the wire. When a pod sends a packet to a service IP address, it is intercepted by iptables which performs DNAT to either the pod or the node's IP address (depending on cluster type). We can see this on our GKE cluster by connecting to the compute engine instance for a node over `ssh` and inspecting its iptables rules.
|
||||
|
||||
```bash
|
||||
gcloud compute ssh --zone 'us-central1-f' 'gke-cluster1-cluster1-pool-9d7804fe-fl8w' --project 'k8s-ip-demo-90bdaee2'
|
||||
@ -265,53 +267,55 @@ KUBE-SVC-MVJGFDRMC5WIL772 tcp -- anywhere 10.107.252.157 /*
|
||||
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
|
||||
```
|
||||
|
||||
But regardless, the end result is the same: Service IP addresses aren't real, so they can be anything. Despite their fictional nature, Google uses a "flat" architecture that does not allow re-using IP addresses across multiple clusters so Google recommends using (TODO ip range) for service IP ranges.
|
||||
But regardless, the end result is the same: Service IP addresses aren't real, so they can be anything. Despite their fictional nature, Google uses a "flat" architecture that does not allow re-using IP addresses across multiple clusters so [Google recommends using slices of `100.64.0.0/10` for service IP ranges](https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-kubernetes-pod-ip-allocation-in-gke).
|
||||
|
||||
Pod IP Addresses
|
||||
----------------
|
||||
|
||||
In the previous section we saw how sending a packet to `service1` results in iptables intercepting that packet and rewriting the destination to the pod IP address. In a VPC-native GKE cluster, each node has a network interface for each pod:
|
||||
Pod IP Addresses, unlike Service IP Addresses, are actually real. We can see this by spinning up a basic http server on the user machine:
|
||||
|
||||
```
|
||||
$ netstat -4nr
|
||||
Kernel IP routing table
|
||||
Destination Gateway Genmask Flags MSS Window irtt Iface
|
||||
0.0.0.0 10.10.10.1 0.0.0.0 UG 0 0 0 eth0
|
||||
10.10.10.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
|
||||
169.254.123.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
|
||||
169.254.169.254 10.10.10.1 255.255.255.255 UGH 0 0 0 eth0
|
||||
240.10.0.2 0.0.0.0 255.255.255.255 UH 0 0 0 gke200305c2a96
|
||||
240.10.0.3 0.0.0.0 255.255.255.255 UH 0 0 0 gke026d556ebe9
|
||||
240.10.0.4 0.0.0.0 255.255.255.255 UH 0 0 0 gke7d4f3a7a7fe
|
||||
240.10.0.5 0.0.0.0 255.255.255.255 UH 0 0 0 gke60f18655088
|
||||
240.10.0.6 0.0.0.0 255.255.255.255 UH 0 0 0 gke1a72a682490
|
||||
240.10.0.8 0.0.0.0 255.255.255.255 UH 0 0 0 gke1c4d51adb0d
|
||||
240.10.0.9 0.0.0.0 255.255.255.255 UH 0 0 0 gke9d25513aa8f
|
||||
240.10.0.10 0.0.0.0 255.255.255.255 UH 0 0 0 gke6a364803b2a
|
||||
240.10.0.12 0.0.0.0 255.255.255.255 UH 0 0 0 gke6a63d89ef86
|
||||
240.10.0.13 0.0.0.0 255.255.255.255 UH 0 0 0 gke35b91a8a487
|
||||
240.10.0.14 0.0.0.0 255.255.255.255 UH 0 0 0 gke96c13f51f03
|
||||
240.10.0.15 0.0.0.0 255.255.255.255 UH 0 0 0 gke84a95b2f8d9
|
||||
240.10.0.16 0.0.0.0 255.255.255.255 UH 0 0 0 gkec88ce3d8bdb
|
||||
240.10.0.17 0.0.0.0 255.255.255.255 UH 0 0 0 gkeacb4e0652ac
|
||||
240.10.0.18 0.0.0.0 255.255.255.255 UH 0 0 0 gke49bb9e75be2
|
||||
240.10.0.19 0.0.0.0 255.255.255.255 UH 0 0 0 gke0ece9ad356b
|
||||
240.10.0.20 0.0.0.0 255.255.255.255 UH 0 0 0 gke0a1351c4ee3
|
||||
240.10.0.21 0.0.0.0 255.255.255.255 UH 0 0 0 gke72a06fc23ca
|
||||
240.10.0.22 0.0.0.0 255.255.255.255 UH 0 0 0 gke9845db36eb5
|
||||
240.10.0.23 0.0.0.0 255.255.255.255 UH 0 0 0 gkecb6bf7230eb
|
||||
240.10.0.24 0.0.0.0 255.255.255.255 UH 0 0 0 gke7dae60021d4
|
||||
240.10.0.25 0.0.0.0 255.255.255.255 UH 0 0 0 gkeb8396784860
|
||||
240.10.0.26 0.0.0.0 255.255.255.255 UH 0 0 0 gke4bd6d44f52d
|
||||
240.10.0.27 0.0.0.0 255.255.255.255 UH 0 0 0 gke3adcfdc91bc
|
||||
240.10.0.28 0.0.0.0 255.255.255.255 UH 0 0 0 gkefabe3212dac
|
||||
240.10.0.29 0.0.0.0 255.255.255.255 UH 0 0 0 gke0f41cfda23e
|
||||
240.10.0.30 0.0.0.0 255.255.255.255 UH 0 0 0 gke91fc0947c42
|
||||
240.10.0.31 0.0.0.0 255.255.255.255 UH 0 0 0 gke9ee620217b1
|
||||
240.10.0.32 0.0.0.0 255.255.255.255 UH 0 0 0 gke12336532836
|
||||
240.10.0.33 0.0.0.0 255.255.255.255 UH 0 0 0 gke369d5150571
|
||||
240.10.0.34 0.0.0.0 255.255.255.255 UH 0 0 0 gke97dfb4bceed
|
||||
240.10.0.35 0.0.0.0 255.255.255.255 UH 0 0 0 gke085b5ff7d93
|
||||
gcloud compute ssh --zone 'us-central1-c' 'user-machine' --project 'k8s-ip-demo-1aa0405a'
|
||||
```
|
||||
|
||||
We can see the pod IP addresses for `service1` of `240.10.0.24` and `240.10.0.25` would route over `gke7dae60021d4` and `gkeb8396784860`. At that point, Google's infrastructure takes over delivering the packet.
|
||||
```
|
||||
python3 -m http.server 8080
|
||||
```
|
||||
|
||||
We can then spin up a pod in our cluster:
|
||||
```
|
||||
kubectl --kubeconfig /bridge/git/kubernetes_ip_demo/output/kubeconfig/cluster1.yaml run --rm -i -t --image alpine:3.21 "testpod-$(uuidgen | cut -d '-' -f 2)" -- /bin/sh
|
||||
```
|
||||
|
||||
and hit the web server on the user machine:
|
||||
```
|
||||
# apk --no-cache add curl
|
||||
# curl http://usermachine.k8sdemo.mydomain.example:8080
|
||||
```
|
||||
|
||||
We will see the output on the `user-machine` `ssh` session:
|
||||
```
|
||||
240.10.1.35 - - [18/Mar/2025 01:00:27] "GET / HTTP/1.1" 200 -
|
||||
```
|
||||
|
||||
But that doesn't mean that we need to use the valuable RFC-1918 IP address space for them. Instead, we can configure our cluster to perform SNAT to the node's IP address using kubernetes' [ip-masq-agent](https://github.com/kubernetes-sigs/ip-masq-agent). This frees us up to use other reserved-but-not-universally-supported IP address ranges like grabbing slices of `240.0.0.0/4`.
|
||||
|
||||
To demonstrate, we can apply the terraform config again but with the `enable_snat=true` variable set:
|
||||
|
||||
```
|
||||
tf apply -var dns_root="k8sdemo.mydomain.example." -var quota_email="MrManager@mydomain.example" -var quota_justification="Explain why you need quotas increased here." -var cluster_exists=true -var enable_snat=true
|
||||
```
|
||||
|
||||
Then in our kubernetes pod, we can run the `curl` again:
|
||||
|
||||
```
|
||||
# curl http://usermachine.k8sdemo.mydomain.example:8080
|
||||
```
|
||||
|
||||
which now shows the node IP address in the `user-machine` console:
|
||||
|
||||
```
|
||||
10.10.10.14 - - [18/Mar/2025 22:43:25] "GET / HTTP/1.1" 200 -
|
||||
```
|
||||
|
||||
So this means that just like Service IP addresses, we can make the pod IP addresses anything. [Google recommends using slices of `` for pod IP ranges](https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-kubernetes-pod-ip-allocation-in-gke), and then enabling SNAT if you need to talk to networks outside of Google Cloud.
|
||||
|
||||
@ -78,6 +78,7 @@ module "cluster1" {
|
||||
external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
cluster_exists = var.cluster_exists
|
||||
enable_snat = var.enable_snat
|
||||
service_container = google_project_service.service["container"]
|
||||
}
|
||||
|
||||
@ -115,6 +116,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -137,6 +139,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -159,6 +162,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -181,6 +185,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -203,6 +208,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -225,6 +231,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -247,6 +254,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -269,6 +277,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -291,6 +300,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -313,6 +323,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -335,6 +346,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -357,6 +369,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -379,6 +392,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -401,6 +415,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -423,6 +438,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -445,6 +461,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -467,6 +484,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -489,6 +507,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -511,6 +530,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -533,6 +553,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -555,6 +576,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -577,6 +599,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -599,6 +622,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -621,6 +645,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -643,6 +668,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -665,6 +691,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -687,6 +714,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -709,6 +737,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -731,6 +760,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -753,6 +783,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -775,6 +806,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -797,6 +829,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -819,6 +852,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -841,6 +875,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -863,6 +898,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -885,6 +921,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -907,6 +944,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -929,6 +967,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -951,6 +990,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -973,6 +1013,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -995,6 +1036,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1017,6 +1059,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1039,6 +1082,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1061,6 +1105,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1083,6 +1128,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1105,6 +1151,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1127,6 +1174,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1149,6 +1197,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1171,6 +1220,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1193,6 +1243,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1215,6 +1266,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1237,6 +1289,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1259,6 +1312,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1281,6 +1335,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1303,6 +1358,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1325,6 +1381,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1347,6 +1404,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1369,6 +1427,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1391,6 +1450,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1413,6 +1473,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1435,6 +1496,7 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -1457,5 +1519,6 @@ output "cluster_ip_address_utilization_url_cluster1" {
|
||||
# external_dns_k8s_service_account = local.external_dns_k8s_service_account
|
||||
# external_dns_gcp_service_account_email = google_service_account.external_dns.email
|
||||
# cluster_exists = var.cluster_exists
|
||||
# enable_snat = var.enable_snat
|
||||
# service_container = google_project_service.service["container"]
|
||||
# }
|
||||
|
||||
@ -73,6 +73,12 @@ variable "ssh_key" {
|
||||
default = null
|
||||
}
|
||||
|
||||
variable "enable_snat" {
|
||||
description = "Whether we should enable source network address translation to the node IP address."
|
||||
type = bool
|
||||
default = false
|
||||
}
|
||||
|
||||
# manual step: enable cloudbilling.googleapis.com in the terraform provider project
|
||||
# https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=terraform-management-427323
|
||||
provider "google" {
|
||||
|
||||
@ -72,7 +72,12 @@ variable "cluster_exists" {
|
||||
variable "routes_based" {
|
||||
description = "Set to true to create a routes-based cluster instead of VPC Native. This is mostly for testing."
|
||||
type = bool
|
||||
default = true
|
||||
default = false
|
||||
}
|
||||
|
||||
variable "enable_snat" {
|
||||
description = "Whether we should enable source network address translation to the node IP address."
|
||||
type = bool
|
||||
}
|
||||
|
||||
output "gke_connect_command" {
|
||||
|
||||
@ -35,5 +35,6 @@ module "workload" {
|
||||
dns_managed_zone = var.dns_managed_zone
|
||||
public_ingress = var.public_ingress
|
||||
ingress_type = var.ingress_type
|
||||
enable_snat = var.enable_snat
|
||||
main_k8s_namespace = var.main_k8s_namespace
|
||||
}
|
||||
|
||||
14
terraform/modules/k8s_workload/ip_masq.tf
Normal file
14
terraform/modules/k8s_workload/ip_masq.tf
Normal file
@ -0,0 +1,14 @@
|
||||
resource "kubernetes_config_map" "ip_masq_agent" {
|
||||
count = var.enable_snat ? 1 : 0
|
||||
|
||||
metadata {
|
||||
name = "ip-masq-agent"
|
||||
namespace = "kube-system"
|
||||
}
|
||||
|
||||
data = {
|
||||
config = "nonMasqueradeCIDRs:\n - 100.64.0.0/19\n - 240.10.0.0/17\nmasqLinkLocal: false\nresyncInterval: 60s\n"
|
||||
}
|
||||
|
||||
depends_on = [var.node_pool]
|
||||
}
|
||||
@ -29,6 +29,11 @@ variable "main_k8s_namespace" {
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "enable_snat" {
|
||||
description = "Whether we should enable source network address translation to the node IP address."
|
||||
type = bool
|
||||
}
|
||||
|
||||
# Provide time for Service cleanup
|
||||
resource "time_sleep" "wait_service_cleanup" {
|
||||
depends_on = [var.cluster]
|
||||
|
||||
@ -19,7 +19,7 @@ resource "google_compute_instance" "user_machine" {
|
||||
name = "user-machine"
|
||||
machine_type = "g1-small"
|
||||
zone = var.zone
|
||||
tags = ["allow-iap-ssh"]
|
||||
tags = ["allow-iap-ssh", "allow-python-http"]
|
||||
|
||||
boot_disk {
|
||||
initialize_params {
|
||||
@ -76,3 +76,16 @@ resource "google_dns_record_set" "user_machine" {
|
||||
|
||||
rrdatas = [google_compute_instance.user_machine.network_interface[0].network_ip]
|
||||
}
|
||||
|
||||
resource "google_compute_firewall" "allow_python_http" {
|
||||
project = google_project.project.project_id
|
||||
name = "allow-python-http"
|
||||
network = google_compute_network.default.id
|
||||
direction = "INGRESS"
|
||||
allow {
|
||||
protocol = "tcp"
|
||||
ports = ["8080"]
|
||||
}
|
||||
source_ranges = ["0.0.0.0/0"]
|
||||
target_tags = ["allow-python-http"]
|
||||
}
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user