subnets | Visual subnet calculator | Apps library

Problem: I am currently using ingress-nginx in my EKS cluster to route traffic to services that need public access.

My use case: I have services I want to deploy in the same cluster but don't want them to have public access. I only want the pods to communicate will all other services within the cluster. Those pods are meant to be private because they're backend services and only need pod-to-pod communication. How do I modify my ingress resource for this purpose?

Cluster Architecture: All services are in the private subnets of the cluster while the load-balancer is in the public subnets

Additional note: I am using external-dns to dynamically create the subdomains for the hosted zones. The hosted zone is public

Thanks

Below are my service.yml and ingress.yml for public services. I want to modify these files for private services

service.yml

I've been trying to get over this but I'm out of ideas for now hence I'm posting the question here.

I'm experimenting with the Oracle Cloud Infrastructure (OCI) and I wanted to create a Kubernetes cluster which exposes some service.

The goal is:

• A running managed Kubernetes cluster (OKE)
• 2 nodes at least
• 1 service that's accessible for external parties

The infra looks the following:

• A VCN for the whole thing
• A private subnet on 10.0.1.0/24
• A public subnet on 10.0.0.0/24
• NAT gateway for the private subnet
• Internet gateway for the public subnet
• Service gateway
• The corresponding security lists for both subnets which I won't share right now unless somebody asks for it
• A containerengine K8S (OKE) cluster in the VCN with public Kubernetes API enabled
• A node pool for the K8S cluster with 2 availability domains and with 2 instances right now. The instances are ARM machines with 1 OCPU and 6GB RAM running Oracle-Linux-7.9-aarch64-2021.12.08-0 images.
• A namespace in the K8S cluster (call it staging for now)
• A deployment which refers to a custom NextJS application serving traffic on port 3000

And now it's the point where I want to expose the service running on port 3000.

I have 2 obvious choices:

• Create a LoadBalancer service in K8S which will spawn a classic Load Balancer in OCI, set up it's listener and set up the backendset referring to the 2 nodes in the cluster, plus it adjusts the subnet security lists to make sure traffic can flow
• Create a Network Load Balancer in OCI and create a NodePort on K8S and manually configure the NLB to the ~same settings as the classic Load Balancer

The first one works perfectly fine but I want to use this cluster with minimal costs so I decided to experiment with option 2, the NLB since it's way cheaper (zero cost).

Long story short, everything works and I can access the NextJS app on the IP of the NLB most of the time but sometimes I couldn't. I decided to look it up what's going on and turned out the NodePort that I exposed in the cluster isn't working how I'd imagine.

The service behind the NodePort is only accessible on the Node that's running the pod in K8S. Assume NodeA is running the service and NodeB is just there chilling. If I try to hit the service on NodeA, everything is fine. But when I try to do the same on NodeB, I don't get a response at all.

That's my problem and I couldn't figure out what could be the issue.

What I've tried so far:

• Switching from ARM machines to AMD ones - no change
• Created a bastion host in the public subnet to test which nodes are responding to requests. Turned out only the node responds that's running the pod.
• Created a regular LoadBalancer in K8S with the same config as the NodePort (in this case OCI will create a classic Load Balancer), that works perfectly
• Tried upgrading to Oracle 8.4 images for the K8S nodes, didn't fix it
• Ran the Node Doctor on the nodes, everything is fine
• Checked the logs of kube-proxy, kube-flannel, core-dns, no error
• Since the cluster consists of 2 nodes, I gave it a try and added one more node and the service was not accessible on the new node either
• Recreated the cluster from scratch

Edit: Some update. I've tried to use a DaemonSet instead of a regular Deployment for the pod to ensure that as a temporary solution, all nodes are running at least one instance of the pod and surprise. The node that was previously not responding to requests on that specific port, it still does not, even though a pod is running on it.

Edit2: Originally I was running the latest K8S version for the cluster (v1.21.5) and I tried downgrading to v1.20.11 and unfortunately the issue is still present.

Edit3: Checked if the NodePort is open on the node that's not responding and it is, at least kube-proxy is listening on it.

I used the vpc module to create my VPC via the following code:

I am trying to create a stack (see code below)

but I get the following error:

When I try to connect to my RDS Postgresql DB I get the following output

Background: We're using AWS Cloud Development Kit (CDK) 2.5.0.

Manually using the AWS Console and hard-coded IP addresses, Route 53 to an ALB (Application Load Balancer) to a private Interface VPC Endpoint to a private REST API-Gateway (and so on..) works. See image below.

Code: We're trying to code this manual solution via CDK, but are stuck on how to get and use the IP addresses or in some way hook up the load balancer to the Interface VPC Endpoint. (Endpoint has 3 IP addresses, one per availability zone in the region.)

The ALB needs a Target Group which targets the IP addresses of the Interface VPC Endpoint. (Using an "instance" approach instead of IP addresses, we tried using InstanceIdTarget with the endpoint's vpcEndpointId, but that failed. We got the error Instance ID 'vpce-WITHWHATEVERWASHERE' is not valid )

Using CDK, we created the following (among other things) using the aws_elasticloadbalancingv2 module:

• ApplicationLoadBalancer (ALB)
• ApplicationTargetGroup (ATG) aka Target Group

We were hopeful about aws_elasticloadbalancingv2_targets similar to aws_route53_targets, but no luck. We know the targets property of the ApplicationTargetGroup takes an array of IApplicationLoadBalancerTarget objects, but that's it.

I am creating nginx ingress controller of type nlb with static ips, but for static ips I am getting this error AllocationIdNotFound. Although this allocation id is valid and eip with this id is present in the same region. Here are the annotations that I am using with nginx ingress controller service

I am working on a Terraform project that has an end goal of an EKS cluster with the following properties:

1. Private to the outside internet
2. Accessible via a bastion host
3. Uses worker groups
4. Resources (deployments, cron jobs, etc) configurable via the Terraform Kubernetes module

To accomplish this, I've modified the Terraform EKS example slightly (code at bottom of the question). The problems that I am encountering is that after SSH-ing into the bastion, I cannot ping the cluster and any commands like kubectl get pods timeout after about 60 seconds.

Here are the facts/things I know to be true:

1. I have (for the time being) switched the cluster to a public cluster for testing purposes. Previously when I had cluster_endpoint_public_access set to false the terraform apply command would not even complete as it could not access the /healthz endpoint on the cluster.
2. The Bastion configuration works in the sense that the user data runs successfully and installs kubectl and the kubeconfig file
3. I am able to SSH into the bastion via my static IP (that's the var.company_vpn_ips in the code)
4. It's entirely possible this is fully a networking problem and not an EKS/Terraform problem as my understanding of how the VPC and its security groups fit into this picture is not entirely mature.

Here is the VPC configuration:

I'm trying to spin up an Aurora Postgres Cluster and I can't seem to make it available over the internet. I'm using Terraform to code the infrastructure.

I've created a security group to allow external access and that is attached to the VPC's subnets used by the Cluster. Still, I can't seem to be able to access the endpoints from my local machine.

I can't figured out what I'm missing.