Explore all Service Mesh open source software, libraries, packages, source code, cloud functions and APIs.

Popular New Releases in Service Mesh

kubernetes-handbook

v20220301

linkerd2

stable-2.11.2

flagger

v1.19.0

meshoptimizer

v0.17

meshlab

MeshLab-2022.02

Popular Libraries in Service Mesh

kubernetes-handbook

by rootsongjc doticonshelldoticon

star image 9910 doticonCC-BY-4.0

Kubernetes中文指南/云原生应用架构实战手册 - https://jimmysong.io/kubernetes-handbook

linkerd2

by linkerd doticongodoticon

star image 8334 doticonApache-2.0

Ultralight, security-first service mesh for Kubernetes. Main repo for Linkerd 2.x.

flagger

by fluxcd doticongodoticon

star image 3572 doticonApache-2.0

Progressive delivery Kubernetes operator (Canary, A/B Testing and Blue/Green deployments)

meshoptimizer

by zeux doticonc++doticon

star image 3049 doticonMIT

Mesh optimization library that makes meshes smaller and faster to render

meshlab

by cnr-isti-vclab doticonc++doticon

star image 2882 doticonGPL-3.0

The open source mesh processing system

kiali

by kiali doticontypescriptdoticon

star image 2805 doticonApache-2.0

Kiali project, observability for the Istio service mesh

kuma

by kumahq doticongodoticon

star image 2711 doticonApache-2.0

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.

osm

by openservicemesh doticongodoticon

star image 2364 doticonApache-2.0

Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.

Deform

by keenanwoodall doticoncsharpdoticon

star image 2315 doticonMIT

A fully-featured deformer system for Unity.

Trending New libraries in Service Mesh

PyMeshLab

by cnr-isti-vclab doticonc++doticon

star image 368 doticonGPL-3.0

The open source mesh processing python library

mesh_to_sdf

by marian42 doticonpythondoticon

star image 344 doticonMIT

Calculate signed distance fields for arbitrary meshes

kubernetes-native-testbed

by kubernetes-native-testbed doticonjavascriptdoticon

star image 239 doticonApache-2.0

This is fully Kubernetes-native testbed environment. Please contribute for add additional OSS (Vitess, NATS, etc) or microservices.

Pose2Mesh_RELEASE

by hongsukchoi doticonpythondoticon

star image 233 doticon

Official Pytorch implementation of "Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose", ECCV 2020

aeraki

by aeraki-framework doticongodoticon

star image 226 doticonApache-2.0

Manage any layer 7 traffic in Istio Service Mesh.

AR_100Days

by satoshi0212 doticonswiftdoticon

star image 203 doticonMIT

IsoMesh

by EmmetOT doticoncsharpdoticon

star image 134 doticonMIT

IsoMesh is a group of related tools for Unity for converting meshes into signed distance field data, raymarching signed distance fields, and extracting signed distance field data back to meshes via surface nets or dual contouring.

MeshDecal

by Fewes doticoncsharpdoticon

star image 118 doticonMIT

A simple mesh decal component for Unity

robust-laplacians-py

by nmwsharp doticonc++doticon

star image 108 doticonMIT

Build high-quality Laplace matrices on meshes and point clouds in Python. Implements [Sharp & Crane SGP 2020].

Top Authors in Service Mesh

1

layer5io

22 Libraries

star icon1481

2

mikolalysenko

18 Libraries

star icon447

3

meshery

12 Libraries

star icon1755

4

substack

8 Libraries

star icon140

5

nschloe

7 Libraries

star icon2397

6

mattatz

6 Libraries

star icon1252

7

istio

6 Libraries

star icon3287

8

kylebarron

5 Libraries

star icon127

9

daviddoria

5 Libraries

star icon17

10

solo-io

4 Libraries

star icon1120

1

22 Libraries

star icon1481

2

18 Libraries

star icon447

3

12 Libraries

star icon1755

4

8 Libraries

star icon140

5

7 Libraries

star icon2397

6

6 Libraries

star icon1252

7

6 Libraries

star icon3287

8

5 Libraries

star icon127

9

5 Libraries

star icon17

10

4 Libraries

star icon1120

Trending Kits in Service Mesh

The use of Java Service mesh libraries like nacos, redisson, gs-rest-service, microservice-istio is a good choice. They are easy to use and they can be integrated with other applications. The reason behind this is that you can use them as a replacement for a legacy application where it will be easier to integrate them into your existing system. This provides a common interface and allows everyone on your team to share information with each other in an easy manner. The only downsides to this approach is that there is no real organization behind these services so they could be replaced by another company at any time which would cause disruption for your application and its users. One option that is gaining popularity is the nacos library. It supports REST and RPC through its own protocol and has built-in support for TLS. As for microservice-istio which we will be using in this article it’s a popular choice among developers who are developing microservices because it provides a lot of features like observability and health monitoring out of the box. It also has built-in support for Docker containers, service discovery by DNS SRV records, container replication and more. These libraries make it easier for you to define the boundaries of your system and then connect them together. It also gives you a lot of flexibility in terms of how you want your system to scale up and down depending on the demand for resources. Popular open source Java Service mesh libraries include

Python Service mesh libraries are used to handle communication between services, in order to guarantee that the required data is always available for the next service request. Python Service mesh libraries like streamlink, service-streamer, meshmash and bert-as-service are used to deploy microservices in a distributed architecture. The Python Service Mesh ecosystem is growing. There are many different tools and libraries available to build your own service mesh. Meshmash uses netlink sockets to allow for an abstraction on top of TCP sockets. Service-streamer uses libtorrent and ZeroMQ as backends for its network protocol. Streamlink uses ZeroMQ as the network protocol and optionally libtorrent as a backend. Service-streamer-as-service is a proxy server based on libtorrent but with additional features like load balancing and capacity planning. Some of the most popular Python Service mesh libraries among developers are

The use of JavaScript Service mesh libraries like rectangular, togetherjs, hypernova, seneca-mesh is a very common practice. This is because they are easy to install and configure. The libraries also make it easier to create applications that can scale to be distributed across the cluster. The service mesh acts as a mediator between services and performs operations like routing messages across multiple services in order to reduce latency and increase reliability. Rectangular is a Node.js library that provides a service mesh for AngularJS applications. It enables you to build distributed applications using JavaScript and Node.js. togetherjs is another open-source service mesh library for JavaScript services, which runs on top of Kubernetes (K8s). It uses the same API as Seneca-Meshes but with some additional features such as automatic configuration and auto-scaling. hypernova is an enterprise grade service mesh framework built on top of Kubernetes (K8s) and Docker. It provides a cloud native API with support for all major languages and frameworks, including Java, Python, Ruby, NodeJS, Go etc. There are several popular open-source JavaScript Service mesh libraries available for developers

The use of Ruby Service mesh libraries like chatwoot, portus, pact-ruby and synapse are becoming more and more important. These libraries provide a lot of features that can be used to build high-performance microservice architecture. Chatwoot is a service mesh library for the Ruby programming language that makes it easy to build and scale distributed applications. Chatwoot is an open source library that provides message queueing between services. It uses Kafka as its message store and offers secure message queueing with SSL encryption, as well as auto-failover and durable messages. Pact-ruby is a Ruby implementation of the Pact protocol for easily communicating with other systems. It also includes support for multiplexing multiple clients on one port and automatically scaling up or down as needed to handle traffic spikes or drops in load. Portus is a service mesh library for the Ruby programming language that makes it easy to build and scale distributed applications.Synapse is the first Ruby Service mesh library to provide an out of box solution for service mesh, allowing developers to focus on business logic rather than infrastructure concerns. Some of the most widely used open source Ruby Service mesh libraries among developers include

C++ Service mesh libraries like cpprestsdk, IncludeOS, evpp, CppMicroServices are the new breed of service mesh that have been emerging over the past few years. These libraries provide high level abstractions and patterns to easily build distributed systems with the help of container based approach. They can be used to build APIs as well as other services in a highly scalable way. The idea behind this approach is to decouple application components from each other and allow them to independently evolve. They have a lot of features available out of the box which you can use without writing any code. IncludeOS is probably the most popular library in this space. It has been around for a while and has been used for many projects. The main reason why it is so popular is because it has good support for multiple languages including C++. There are also other libraries like cppmicroservices which you can use if you want to build your own service mesh from scratch. Popular open source C++ Service mesh libraries for developers include

Go service mesh libraries like istio, linkerd2, osm, gloo-mesh are used to introduce the concept of microservices in Go. These libraries can provide end-to-end visibility of services across the cluster. In addition, they also provide a number of other features such as load balancing and service discovery which helps you to scale your applications easily. Linkerd2 is the main service mesh library used by Linkerd and many other companies using Linkerd as their primary service mesh. iGloobus Mesh is another option, though it was developed for Google Cloud Platform (GCP) specifically. Gloobus Mesh can be configured to work with multiple clouds. Istio is also an excellent choice for service mesh, but its main advantage over Linkerd is that it’s built on Kubernetes and runs on any cloud provider that supports Kubernetes. As such, it can be used with any orchestrator — including Kubernetes itself — and works with all the major cloud providers. Developers tend to use some of the following open source Go service mesh libraries

PHP Service mesh libraries are used to connect different pieces of code together, so they can interact with each other. These libraries can help you build scalable applications that can communicate with each other through a network of services. PHP Service mesh libraries like monolog, wave, organizr, phpbu are very popular and have a lot of features to help you build your application. They are also very easy to use and can be easily integrated into your project. They have better performance due to their built-in caching mechanism which is not available in the other libraries. Also they provide better support for multiple languages. Monolog is the most popular library for monitoring your applications and services. It is a very lightweight library that can be used to monitor almost anything in your application. Monolog has built-in support for HTTP, SQL, Redis, and web sockets. Wave uses a different approach to monitoring by taking advantage of Kubernetes resources and services. It allows you to manage your pods as well as scale down or scale up your applications based on load conditions. The Organizr library is made up of several components, each one dealing with a specific aspect of monitoring an application or service. For example, there is an agent component that runs on each node and an agent manager that monitors all nodes in your cluster. The agent manager sends notifications to other components when alerts occur, such as if there are errors in the logs or if something goes wrong with a service. There are several popular open source PHP Service mesh libraries available for developers

C# Service mesh libraries are a new way of building microservices. The idea is that you build your entire application as a series of services, and then use a service mesh to connect them together. The most common are Nancy and Winsw. But there are others like Hangfire and ServiceStack that have been gaining traction lately. One is not better than the other, but it's good to know what all the options are so you can make an informed decision about which one to use. Nancy is an open source project that provides a framework for monitoring microservices and orchestrating applications as they scale up or down in size. It provides features such as health checking and auto-discovery of services in your system. It also supports reactive messaging between client applications and services, which makes it a good choice if you want to build distributed systems with asynchronous communication patterns. Winsw is an open source service mesh built on Azure Service Fabric (formerly known as Project Natick). It's designed to provide scalable, reliable, elastic and resilient messaging infrastructure for modern cloud applications. Winsw provides high availability for your microservices by maintaining stateful endpoints for each individual service instance with its own set of routing rules in one place. A few of the most popular open source C# Service mesh libraries for developers are

Trending Discussions on Service Mesh

Is 'No Workload identity for a node level' or 'failure to load CA secret' stopping service mesh from working?

Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

Spring boot actuator metrics for Prometheus in Consul Connect

How to create circuit breaker for cloud run services?

Why maxRequestPerConnection of istio does effect to http/1.1 requests?

How to implement role-based auth with SPIFFE/SPIRE?

Accessing an SMTP server when istio is enabled

What is the difference between ingress and service mesh in kubernetes?

How to specify custom Istio ingress gateway in Kubernetes ingress

AWS EKS: unable to attach IAM role to pods

QUESTION

Is 'No Workload identity for a node level' or 'failure to load CA secret' stopping service mesh from working?

Asked 2022-Mar-23 at 17:04

This is the first time I have been trying to install managed Anthos into one of the clusters in GKE. I admit I do not fully understand the full process of installation and troubleshooting I have already done.

It looks like a managed service has failed to install. When I run:

1kubectl describe controlplanerevision asm-managed -n istio-system
2

I get this status:

1kubectl describe controlplanerevision asm-managed -n istio-system
2Status:
3  Conditions:
4    Last Transition Time:  2022-03-15T14:16:21Z
5    Message:               The provisioning process has not completed successfully
6    Reason:                NotProvisioned
7    Status:                False
8    Type:                  Reconciled
9    Last Transition Time:  2022-03-15T14:16:21Z
10    Message:               Provisioning has finished
11    Reason:                ProvisioningFinished
12    Status:                True
13    Type:                  ProvisioningFinished
14    Last Transition Time:  2022-03-15T14:16:21Z
15    Message:               Workload identity is not enabled at node level
16    Reason:                PreconditionFailed
17    Status:                True
18    Type:                  Stalled
19Events:                    <none>
20

However, I have Workload identity enabled on a cluster level and I cannot see any options in GCP Console to set that for just a node level.

enter image description here

I am not sure if this is related to istiod-asm-1125-0 logging some errors. One of them is about failure to load CA secret:

enter image description here

Nevertheless, the service mesh does not show as added or connected in Anthos Dashboard. The cluster is registered with Anthos.enter image description here

ANSWER

Answered 2022-Mar-23 at 17:04
  1. I created a new node pool with more CPU and more nodes as I was getting warning about not having enough CPU. Istio service mesh increases the need for CPU.

  2. I migrated my deployment from old node pool to the new one.

  3. I run istioctl analyze -A and found a few warnings about istio-injection not being enabled in a few namespaces. I fixed that.

  4. I re run asmcli install command without CA

./asmcli install --project_id my-app --cluster_name my-cluster --cluster_location europe-west1-b --fleet_id my-app --output_dir anthos-service-mesh --enable_all

All or some of the above did the trick.

Source https://stackoverflow.com/questions/71496152

QUESTION

Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

Asked 2022-Mar-21 at 08:25

I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.

My setup:

  • ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: my-etcd-cluster.my-etcd-namespace.svc.cluster.local)
  • I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
  • I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD

Here's my affected Go code:

1cli, err := clientv3.New(clientv3.Config{
2    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
3    DialTimeout: 5 * time.Second,
4    Username:    username,
5    Password:    password,
6})
7
8if err != nil {
9    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
10}
11

And here's an error I'm getting when clientv3.New(...) gets executed:

1cli, err := clientv3.New(clientv3.Config{
2    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
3    DialTimeout: 5 * time.Second,
4    Username:    username,
5    Password:    password,
6})
7
8if err != nil {
9    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
10}
11{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
12"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
13"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
14...
151.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
161.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
17sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
18        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
19sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
20        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
21

The same error happens when I'm passing some dummy, invalid credentials.

However, when I tried to access the database in a HTTP API manner:

1cli, err := clientv3.New(clientv3.Config{
2    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
3    DialTimeout: 5 * time.Second,
4    Username:    username,
5    Password:    password,
6})
7
8if err != nil {
9    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
10}
11{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
12"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
13"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
14...
151.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
161.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
17sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
18        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
19sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
20        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
21postBody, _ := json.Marshal(map[string]string{
22    "name":     username,
23    "password": password,
24})
25responseBody := bytes.NewBuffer(postBody)
26
27resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
28if err != nil {
29    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
30}
31l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
32defer resp.Body.Close()
33

...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.

When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.

EDIT: I've also added a simple pod to try accessing my etcd db via etcdctl:

1cli, err := clientv3.New(clientv3.Config{
2    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
3    DialTimeout: 5 * time.Second,
4    Username:    username,
5    Password:    password,
6})
7
8if err != nil {
9    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
10}
11{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
12"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
13"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
14...
151.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
161.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
17sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
18        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
19sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
20        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
21postBody, _ := json.Marshal(map[string]string{
22    "name":     username,
23    "password": password,
24})
25responseBody := bytes.NewBuffer(postBody)
26
27resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
28if err != nil {
29    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
30}
31l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
32defer resp.Body.Close()
33apiVersion: v1
34kind: Pod
35metadata:
36  name: test-pod
37  namespace: my-controller-namespace
38spec:
39  containers:
40  - name: etcdctl
41    image: bitnami/etcd
42    command:
43    - sleep
44    - infinity
45

When logged into the container via kubectl exec, I was able to access my db:

1cli, err := clientv3.New(clientv3.Config{
2    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
3    DialTimeout: 5 * time.Second,
4    Username:    username,
5    Password:    password,
6})
7
8if err != nil {
9    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
10}
11{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
12"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
13"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
14...
151.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
161.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
17sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
18        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
19sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
20        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
21postBody, _ := json.Marshal(map[string]string{
22    "name":     username,
23    "password": password,
24})
25responseBody := bytes.NewBuffer(postBody)
26
27resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
28if err != nil {
29    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
30}
31l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
32defer resp.Body.Close()
33apiVersion: v1
34kind: Pod
35metadata:
36  name: test-pod
37  namespace: my-controller-namespace
38spec:
39  containers:
40  - name: etcdctl
41    image: bitnami/etcd
42    command:
43    - sleep
44    - infinity
45$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
46OK
47

I guess the problem is somewhere in the SDK?

ANSWER

Answered 2022-Mar-21 at 08:25

Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0. As seen in ETCD changelog (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md):

enter image description here

Source https://stackoverflow.com/questions/71509351

QUESTION

Spring boot actuator metrics for Prometheus in Consul Connect

Asked 2022-Feb-18 at 06:48

I have a spring boot application running in a Nomad cluster with Consul Connect enabled.

1network {
2  mode = "bridge"
3}
4
5service {
6  name = "api"
7  port = "9966"
8
9  connect {
10    sidecar_service {}
11  }
12}
13

There is no port mapping defined and the API is reachable only within consul service mesh through the proxy. Now I have prometheus running in the same cluster. How does prometheus discover the individual API instances and scrape metrics out of it. I used the below config. But without having a mapping port in the host level, it is not able to reach the individual API instance.

1network {
2  mode = "bridge"
3}
4
5service {
6  name = "api"
7  port = "9966"
8
9  connect {
10    sidecar_service {}
11  }
12}
13- job_name: 'actuator'
14    metrics_path: /api/actuator/prometheus
15    consul_sd_configs:
16    - server: '{{ env "NOMAD_IP_prometheus_ui" }}:8500'
17      services: ['api']
18

How to solve this problem? What is general practice to scrape metrics from a spring boot application running inside a service mesh with no host port mapping?

ANSWER

Answered 2022-Feb-18 at 06:48

Finally found it. Nomad has an option to expose a particular endpoint via sidecar proxy without mTLS authentication. The use case of this option is specifically for health check or metrics.

https://www.nomadproject.io/docs/job-specification/expose#expose-examples

The expose stanza inside connect stanza helps to achieve this.

1network {
2  mode = "bridge"
3}
4
5service {
6  name = "api"
7  port = "9966"
8
9  connect {
10    sidecar_service {}
11  }
12}
13- job_name: 'actuator'
14    metrics_path: /api/actuator/prometheus
15    consul_sd_configs:
16    - server: '{{ env "NOMAD_IP_prometheus_ui" }}:8500'
17      services: ['api']
18connect {
19        sidecar_service {
20          proxy {
21            expose {
22              path {
23                path             =  "/actuator/prometheus"
24                protocol         =  "http"
25                local_path_port  =  9966
26                listener_port    =  "metrics"
27              }
28            }
29          }
30        }
31      }
32

Source https://stackoverflow.com/questions/69218988

QUESTION

How to create circuit breaker for cloud run services?

Asked 2022-Jan-30 at 15:53

I am trying to understand how we can create circuit breakers for cloud run services,Unlike in GKE we are using istio kind of service mesh how we implement same thing cloud Run ?

ANSWER

Answered 2022-Jan-30 at 15:53

On GKE you'd set up a circuit breaker to prevent overloading your legacy backend systems from a surge in requests.

To accomplish the same on Cloud Run or Cloud Functions, you can set a maximum number of instances. From that documentation:

Specifying maximum instances in Cloud Run allows you to limit the scaling of your service in response to incoming requests, although this maximum setting can be exceeded for a brief period due to circumstances such as traffic spikes. Use this setting as a way to control your costs or to limit the number of connections to a backing service, such as to a database.

Source https://stackoverflow.com/questions/70914326

QUESTION

Why maxRequestPerConnection of istio does effect to http/1.1 requests?

Asked 2021-Nov-04 at 09:19

I'm just learning service mesh using istio and I found a strange behavior. To understand maxRequestsPerConnection of Istio DestinationRule CRD, I write the below manifest and apply it.

1apiVersion: networking.istio.io/v1alpha3
2kind: DestinationRule
3metadata:
4  name: httpbin
5spec:
6  host: httpbin
7  trafficPolicy:
8    connectionPool:
9      tcp:
10        maxConnections: 1
11      http:
12        http1MaxPendingRequests: 1
13        maxRequestsPerConnection: 1
14

And then, I sent requests using fortio. The result is below:

1apiVersion: networking.istio.io/v1alpha3
2kind: DestinationRule
3metadata:
4  name: httpbin
5spec:
6  host: httpbin
7  trafficPolicy:
8    connectionPool:
9      tcp:
10        maxConnections: 1
11      http:
12        http1MaxPendingRequests: 1
13        maxRequestsPerConnection: 1
14yunoMacBook-Air:labo8 yu$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 1000 -loglevel Error http://httpbin:8000/get
1507:12:01 I logger.go:127> Log level is now 4 Error (was 2 Info)
16Fortio 1.11.3 running at 0 queries per second, 2->2 procs, for 1000 calls: http://httpbin:8000/get
17Aggregated Function Time : count 1000 avg 0.0036879818 +/- 0.004588 min 0.000379697 max 0.034176044 sum 3.68798183
18# target 50% 0.00234783
19# target 75% 0.0032551
20# target 90% 0.008
21# target 99% 0.025
22# target 99.9% 0.032784
23Sockets used: 876 (for perfect keepalive, would be 5)
24Jitter: false
25Code 200 : 126 (12.6 %)
26Code 503 : 874 (87.4 %)
27All done 1000 calls (plus 0 warmup) 3.688 ms avg, 1170.1 qps
28yunoMacBook-Air:labo8 yu$
29

After that, I changed maxRequestsPerConnection value to 10:

1apiVersion: networking.istio.io/v1alpha3
2kind: DestinationRule
3metadata:
4  name: httpbin
5spec:
6  host: httpbin
7  trafficPolicy:
8    connectionPool:
9      tcp:
10        maxConnections: 1
11      http:
12        http1MaxPendingRequests: 1
13        maxRequestsPerConnection: 1
14yunoMacBook-Air:labo8 yu$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 1000 -loglevel Error http://httpbin:8000/get
1507:12:01 I logger.go:127> Log level is now 4 Error (was 2 Info)
16Fortio 1.11.3 running at 0 queries per second, 2->2 procs, for 1000 calls: http://httpbin:8000/get
17Aggregated Function Time : count 1000 avg 0.0036879818 +/- 0.004588 min 0.000379697 max 0.034176044 sum 3.68798183
18# target 50% 0.00234783
19# target 75% 0.0032551
20# target 90% 0.008
21# target 99% 0.025
22# target 99.9% 0.032784
23Sockets used: 876 (for perfect keepalive, would be 5)
24Jitter: false
25Code 200 : 126 (12.6 %)
26Code 503 : 874 (87.4 %)
27All done 1000 calls (plus 0 warmup) 3.688 ms avg, 1170.1 qps
28yunoMacBook-Air:labo8 yu$
29apiVersion: networking.istio.io/v1alpha3
30kind: DestinationRule
31metadata:
32  name: httpbin
33spec:
34  (...)
35        maxRequestsPerConnection: 10
36

and I sent requests again with the same settings.

1apiVersion: networking.istio.io/v1alpha3
2kind: DestinationRule
3metadata:
4  name: httpbin
5spec:
6  host: httpbin
7  trafficPolicy:
8    connectionPool:
9      tcp:
10        maxConnections: 1
11      http:
12        http1MaxPendingRequests: 1
13        maxRequestsPerConnection: 1
14yunoMacBook-Air:labo8 yu$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 1000 -loglevel Error http://httpbin:8000/get
1507:12:01 I logger.go:127> Log level is now 4 Error (was 2 Info)
16Fortio 1.11.3 running at 0 queries per second, 2->2 procs, for 1000 calls: http://httpbin:8000/get
17Aggregated Function Time : count 1000 avg 0.0036879818 +/- 0.004588 min 0.000379697 max 0.034176044 sum 3.68798183
18# target 50% 0.00234783
19# target 75% 0.0032551
20# target 90% 0.008
21# target 99% 0.025
22# target 99.9% 0.032784
23Sockets used: 876 (for perfect keepalive, would be 5)
24Jitter: false
25Code 200 : 126 (12.6 %)
26Code 503 : 874 (87.4 %)
27All done 1000 calls (plus 0 warmup) 3.688 ms avg, 1170.1 qps
28yunoMacBook-Air:labo8 yu$
29apiVersion: networking.istio.io/v1alpha3
30kind: DestinationRule
31metadata:
32  name: httpbin
33spec:
34  (...)
35        maxRequestsPerConnection: 10
36yunoMacBook-Air:labo8 yu$ kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 1000 -loglevel Error http://httpbin:8000/get
3707:11:07 I logger.go:127> Log level is now 4 Error (was 2 Info)
38Fortio 1.11.3 running at 0 queries per second, 2->2 procs, for 1000 calls: http://httpbin:8000/get
39Aggregated Function Time : count 1000 avg 0.0039736575 +/- 0.004068 min 0.000404827 max 0.030141552 sum 3.97365754
40# target 50% 0.00231923
41# target 75% 0.00475
42# target 90% 0.0104667
43# target 99% 0.0192
44# target 99.9% 0.025
45Sockets used: 723 (for perfect keepalive, would be 5)
46Jitter: false
47Code 200 : 281 (28.1 %)
48Code 503 : 719 (71.9 %)
49All done 1000 calls (plus 0 warmup) 3.974 ms avg, 1098.3 qps
50yunoMacBook-Air:labo8 yu$
51

200 rate increased and I cannot understand why it happened. In my understanding, fortio uses http/1.1 and only one HTTP request is in one TCP connection when I use http/1.1. So I expected that I get the same results.

Could you tell me why this happened?

ANSWER

Answered 2021-Nov-03 at 09:35

First things first: HTTP/1.1 does allow multiple request per connection with Keep-Alive header. This is the default behavior (RFC 2616, Section 8.1).


The documentation is a bit unclear.

maxRequestsPerConnection description states:

Maximum number of requests per connection to a backend. Setting this parameter to 1 disables keep alive. Default 0, meaning “unlimited”, up to 2^29.

Setting maxRequestsPerConnection to 1 disables Keep-Alive. Setting it to any other value (value > 1) switches Keep-Alive back on.

Setting this field to proper value (not too high, not too low) is the hard part of configuring Istio, and is dependent on your application needs and traffic.

Source https://stackoverflow.com/questions/69777668

QUESTION

How to implement role-based auth with SPIFFE/SPIRE?

Asked 2021-Sep-24 at 00:45

I'm in the process of vetting a move to service mesh. While Istio and Consul Connect are certainly still in the cards, I'm leaning towards building up from a bit lower level with Linkerd and SPIFFE/SPIRE.

I want to build a 'hello world' mesh to test this architecture out. In this hello world mesh, I'd like to be able to issue certificates from SPIFFE/SPIRE that encode some kind of role. As you can probably tell, I'm new to service meshes. How would I implement roles? Are there any guides out there to help get me started?

ANSWER

Answered 2021-Sep-23 at 16:25

So there isn't any kind of integration with Linkerd and SPIFFE/SPIRE. As of right now there isn't any particular plan for an integration but you can see the existing issue for more details. The next release of Linkerd, 2.11, will include the ability to create server side policy but current versions don't enforce any kind of policy settings.

Source https://stackoverflow.com/questions/69291974

QUESTION

Accessing an SMTP server when istio is enabled

Asked 2021-Sep-16 at 12:57

getting error curl: (56) response reading failed while trying to send email via smtp using curl. checked the isto-proxy logs of sidecar but don't see any error logs related to this host. Tried the solution mentioned in How to access external SMTP server from within Kubernetes cluster with Istio Service Mesh as well but didn't work.

service entry

1apiVersion: networking.istio.io/v1beta1
2kind: ServiceEntry
3metadata:
4  name: smtp
5spec:
6  addresses:
7  - 192.168.8.45/32
8  hosts:
9  - smtp.example.com"
10  location: MESH_EXTERNAL
11  ports:
12    - name: tcp-smtp
13      number: 2255
14      protocol: TCP
15

ANSWER

Answered 2021-Sep-14 at 10:38

Most probably port number is causing the error and if not, try deleting the mesh policies

Also please validate based on below points:

1.If you recently updated istio try downgrading it. 2.Look again in Sidecar logs for any conflicts or try disabling it. 3.When it comes to curl 56 error packet transmission; limit could be the problem.

Source https://stackoverflow.com/questions/69164089

QUESTION

What is the difference between ingress and service mesh in kubernetes?

Asked 2021-Aug-31 at 15:33

Can someone help me to understand if service mesh itself is a type of ingress or if there is any difference between service mesh and ingress?

ANSWER

Answered 2021-Aug-31 at 10:45

An "Ingress" is responsible for Routing Traffic into your Cluster (from the Docs: An API object that manages external access to the services in a cluster, typically HTTP.)

On the other side, a Service-Mesh is a tool that adds proxy-Containers as Sidecars to your Pods and Routs traffic between your Pods through those proxy-Containers.

use-Cases for Service-Meshes are i.E.

  • distributed tracing
  • secure (SSL) connections between pods
  • resilience (service-mesh can reroute traffic from failed requests)
  • network-performance-monitoring

Source https://stackoverflow.com/questions/68995087

QUESTION

How to specify custom Istio ingress gateway in Kubernetes ingress

Asked 2021-Aug-19 at 07:33

I deployed Istio using the operator and added a custom ingress gateway which is only accessible from a certain source range (our VPN).

1apiVersion: install.istio.io/v1alpha1
2kind: IstioOperator
3metadata:
4  namespace: istio-system
5  name: ground-zero-ingressgateway
6spec:
7  profile: empty
8  components:
9    ingressGateways:
10      - name: istio-ingressgateway
11        enabled: true
12      - name: istio-vpn-ingressgateway
13        label:
14          app: istio-vpn-ingressgateway
15          istio: vpn-ingressgateway
16        enabled: true
17        k8s:
18          serviceAnnotations:
19            ...
20          service:
21            loadBalancerSourceRanges:
22              - "x.x.x.x/x"
23

Now I want to configure Istio to expose a service outside of the service mesh cluster, using the Kubernetes Ingress resource. I use the kubernetes.io/ingress.class annotation to tell the Istio gateway controller that it should handle this Ingress.

1apiVersion: install.istio.io/v1alpha1
2kind: IstioOperator
3metadata:
4  namespace: istio-system
5  name: ground-zero-ingressgateway
6spec:
7  profile: empty
8  components:
9    ingressGateways:
10      - name: istio-ingressgateway
11        enabled: true
12      - name: istio-vpn-ingressgateway
13        label:
14          app: istio-vpn-ingressgateway
15          istio: vpn-ingressgateway
16        enabled: true
17        k8s:
18          serviceAnnotations:
19            ...
20          service:
21            loadBalancerSourceRanges:
22              - "x.x.x.x/x"
23apiVersion: networking.k8s.io/v1
24kind: Ingress
25metadata:
26  name: my-ingress
27  annotations:
28    kubernetes.io/ingress.class: istio
29spec:
30   ...
31
  • Kubernetes version (EKS): 1.19
  • Istio version: 1.10.3

Which ingress gateway controller is now used (istio-ingressgateway or istio-vpn-ingressgateway)? Is there a way to specify which one should be used?

P.S. I know that I could create a VirtualService and specify the correct gateway but we want to write a manifest that also works without Istio by specifying the correct ingress controller with an annotation.

ANSWER

Answered 2021-Aug-19 at 07:33

You can create an ingress class that references the ingress controller that is deployed by default in the istio-system namespace. This configuration with ingress will work, however to my current knowledge, this is only used for backwards compatibility. If you want to use istio ingress controller functionality, you should use istio gateway and virtual service instead:

Using the Istio Gateway, rather than Ingress, is recommended to make use of the full feature set that Istio offers, such as rich traffic management and security features.

If this solution is not optimal for you, you should use e.g. nginx ingress controller and you can still bind it with annotations (deprecated) or using IngressClass. To my present knowledge it is not possible to bind this ingress class with an additional ingress controller. If you need an explanation, documentation, you should create an issue on github.

Summary: The recommended option is to use the gateway with virtual service. Another possibility is to use nginx alone ingress with different classes and an ingress resource for them.

Source https://stackoverflow.com/questions/68633656

QUESTION

AWS EKS: unable to attach IAM role to pods

Asked 2021-Aug-12 at 14:55

So i created an AWS EKS cluster & proceeded with trying to created a service mesh using AWS App Mesh on AWS EKS using EKS workshop & AWS App Mesh user guide. The appmesh controller installs.

kubectl get pods confirms it.

1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3

Then did the following -

  1. create a namespace & mesh (following user guide). Used following YAML config -
1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22
  1. create IAM service account. kubectl describe for the service account returns this.
1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22Name:                example-svc-acct
23Namespace:           example
24Labels:              <none>
25Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/eksctl-eks-addon-iamserviceaccount-example-Role1
26Image pull secrets:  <none>
27Mountable secrets:   example-svc-acct-token-lgrs2
28Tokens:              example-svc-acct-token-lgrs2
29Events:              <none>
30

I can see the required annotation as per this 3. I deploy my service using helm. kubectl get pods -n example shows

1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22Name:                example-svc-acct
23Namespace:           example
24Labels:              <none>
25Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/eksctl-eks-addon-iamserviceaccount-example-Role1
26Image pull secrets:  <none>
27Mountable secrets:   example-svc-acct-token-lgrs2
28Tokens:              example-svc-acct-token-lgrs2
29Events:              <none>
30NAME                      READY   STATUS    RESTARTS   AGE
31svc1-5d4b4d6485-m7t7g      1/2     Running   0          7s
32svc2-76cb5fd545-nqgx5      2/3     Running   0          7s
33svc2-76cb5fd545-vsbnj      2/3     Running   0          7s
34svc3-84f97bd64f-q9hjx      1/2     Running   0          7s
35

The envoy container is unable to move to ready state.

  1. Looking for environment variables in the container shows missing variables
1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22Name:                example-svc-acct
23Namespace:           example
24Labels:              <none>
25Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/eksctl-eks-addon-iamserviceaccount-example-Role1
26Image pull secrets:  <none>
27Mountable secrets:   example-svc-acct-token-lgrs2
28Tokens:              example-svc-acct-token-lgrs2
29Events:              <none>
30NAME                      READY   STATUS    RESTARTS   AGE
31svc1-5d4b4d6485-m7t7g      1/2     Running   0          7s
32svc2-76cb5fd545-nqgx5      2/3     Running   0          7s
33svc2-76cb5fd545-vsbnj      2/3     Running   0          7s
34svc3-84f97bd64f-q9hjx      1/2     Running   0          7s
35kubectl exec -n example svc3-84f97bd64f-q9hjx -c envoy env | grep AWS
36AWS_REGION=us-east-2
37

As per docs, these AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN should have been there.

  1. kubectl logs for envoy container shows permission problems
1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22Name:                example-svc-acct
23Namespace:           example
24Labels:              <none>
25Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/eksctl-eks-addon-iamserviceaccount-example-Role1
26Image pull secrets:  <none>
27Mountable secrets:   example-svc-acct-token-lgrs2
28Tokens:              example-svc-acct-token-lgrs2
29Events:              <none>
30NAME                      READY   STATUS    RESTARTS   AGE
31svc1-5d4b4d6485-m7t7g      1/2     Running   0          7s
32svc2-76cb5fd545-nqgx5      2/3     Running   0          7s
33svc2-76cb5fd545-vsbnj      2/3     Running   0          7s
34svc3-84f97bd64f-q9hjx      1/2     Running   0          7s
35kubectl exec -n example svc3-84f97bd64f-q9hjx -c envoy env | grep AWS
36AWS_REGION=us-east-2
37[2021-08-02 22:07:12.516][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
38[2021-08-02 22:07:16.268][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example
39[2021-08-02 22:07:21.402][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
40[2021-08-02 22:07:42.125][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
41

The role attached to service account has action appmesh:StreamAggregatedResources permitted on all resources.

I can see the problem in step 3. Having looked in different places for an entire day I cannot figure out what I am missing to get the required role attached to the container, and thus set the needed environment variables.

Any pointer will be appreciated. Thanks.

More info:

1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22Name:                example-svc-acct
23Namespace:           example
24Labels:              <none>
25Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/eksctl-eks-addon-iamserviceaccount-example-Role1
26Image pull secrets:  <none>
27Mountable secrets:   example-svc-acct-token-lgrs2
28Tokens:              example-svc-acct-token-lgrs2
29Events:              <none>
30NAME                      READY   STATUS    RESTARTS   AGE
31svc1-5d4b4d6485-m7t7g      1/2     Running   0          7s
32svc2-76cb5fd545-nqgx5      2/3     Running   0          7s
33svc2-76cb5fd545-vsbnj      2/3     Running   0          7s
34svc3-84f97bd64f-q9hjx      1/2     Running   0          7s
35kubectl exec -n example svc3-84f97bd64f-q9hjx -c envoy env | grep AWS
36AWS_REGION=us-east-2
37[2021-08-02 22:07:12.516][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
38[2021-08-02 22:07:16.268][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example
39[2021-08-02 22:07:21.402][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
40[2021-08-02 22:07:42.125][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
41$ eksctl version
420.42.0
43$ kubectl version
44Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.11-eks-cfdc40", GitCommit:"cfdc40d4c1b7d14eb60152107963ae41aa2e4804", GitTreeState:"clean", BuildDate:"2020-09-17T17:10:39Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
45Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
46

ANSWER

Answered 2021-Aug-12 at 14:55

Apparently, it was a stupid mistake of missing out serviceAccountName in the deployment template spec.

1NAMESPACE        NAME                                            READY   STATUS    RESTARTS   AGE
2appmesh-system   appmesh-controller-847f957bc8-s2k7l             1/1     Running   0          57m
3apiVersion: v1
4kind: Namespace
5metadata:
6  name: example
7  labels:
8    mesh: v-mesh
9    gateway: ingress-gw
10    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
11---
12apiVersion: appmesh.k8s.aws/v1beta2
13kind: Mesh
14metadata:
15  name: v-mesh
16spec:
17  namespaceSelector:
18    matchLabels:
19      mesh: v-mesh
20  egressFilter:
21    type: ALLOW_ALL
22Name:                example-svc-acct
23Namespace:           example
24Labels:              <none>
25Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx:role/eksctl-eks-addon-iamserviceaccount-example-Role1
26Image pull secrets:  <none>
27Mountable secrets:   example-svc-acct-token-lgrs2
28Tokens:              example-svc-acct-token-lgrs2
29Events:              <none>
30NAME                      READY   STATUS    RESTARTS   AGE
31svc1-5d4b4d6485-m7t7g      1/2     Running   0          7s
32svc2-76cb5fd545-nqgx5      2/3     Running   0          7s
33svc2-76cb5fd545-vsbnj      2/3     Running   0          7s
34svc3-84f97bd64f-q9hjx      1/2     Running   0          7s
35kubectl exec -n example svc3-84f97bd64f-q9hjx -c envoy env | grep AWS
36AWS_REGION=us-east-2
37[2021-08-02 22:07:12.516][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
38[2021-08-02 22:07:16.268][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example
39[2021-08-02 22:07:21.402][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
40[2021-08-02 22:07:42.125][1][warning][config] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:101] StreamAggregatedResources gRPC config stream closed: 7, Unauthorized to perform appmesh:StreamAggregatedResources for arn:aws:appmesh:us-east-2:xxxxx:mesh/v-mesh/virtualNode/svc3-vn_example.
41$ eksctl version
420.42.0
43$ kubectl version
44Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.11-eks-cfdc40", GitCommit:"cfdc40d4c1b7d14eb60152107963ae41aa2e4804", GitTreeState:"clean", BuildDate:"2020-09-17T17:10:39Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
45Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
46spec:
47  serviceAccountName: {{ .Values.serviceAccount.name }}
48

Added that & the problem went away.

Source https://stackoverflow.com/questions/68628490

Community Discussions contain sources that include Stack Exchange Network

Tutorials and Learning Resources in Service Mesh

Tutorials and Learning Resources are not available at this moment for Service Mesh

Share this Page

share link

Get latest updates on Service Mesh