kandi background
kandi background
Explore Kits
kandi background
Explore Kits
kandi background
Explore Kits
kandi background
Explore Kits
Explore all Microservice open source software, libraries, packages, source code, cloud functions and APIs.

Popular New Releases in Microservice

v1.0.1

Istio 1.13.3

Apollo 2.0.0-RC1 Release

Apollo 1.9.0 Release

2.1.0-BETA (Apr 01, 2022)

mall

v1.0.1

istio

Istio 1.13.3

apollo

Apollo 2.0.0-RC1 Release

apollo

Apollo 1.9.0 Release

nacos

2.1.0-BETA (Apr 01, 2022)

Popular Libraries in Microservice

Trending New libraries in Microservice

Top Authors in Microservice

1

63 Libraries

2162

2

38 Libraries

1948

3

26 Libraries

2155

4

25 Libraries

16873

5

23 Libraries

1078

6

22 Libraries

1437

7

20 Libraries

186

8

20 Libraries

153

9

19 Libraries

2443

10

16 Libraries

5847

1

63 Libraries

2162

2

38 Libraries

1948

3

26 Libraries

2155

4

25 Libraries

16873

5

23 Libraries

1078

6

22 Libraries

1437

7

20 Libraries

186

8

20 Libraries

153

9

19 Libraries

2443

10

16 Libraries

5847

Trending Kits in Microservice

go-microservice

10 best Go Microservice

Go microservices are small, single-purpose services that can be independently deployed and scaled. This approach is often used in applications that need to scale well, such as production systems or web applications. One of the biggest advantages of using Go microservices libraries is that you get better performance and reliability than you would by writing your own code. You also avoid the maintenance overhead of managing a monolithic application. Kratos is an open source library that implements Erlang actor framework in Go. Actor-model based systems are one of the most popular patterns for building distributed systems. Actors are lightweight processes that communicate with each other via message passing, which makes them very scalable and durable. Kratos provides an implementation of Erlang actors for Go so you can easily build high performance, reliable systems on top of it. Nomad is another open source library that implements Erlang actor framework in Go and works with kubectl to manage your cluster from within your program code. Nomad supports all kinds of clusters (mock, test) but its main goal is to provide a consistent interface between operations that want a cluster (e.g., Kubernetes). Popular open source Go microservice libraries include:

csharp-microservice

11 best C# Microservice

C# Microservice libraries are a way to break up large applications into smaller pieces that are easier to build and maintain. They offer many benefits in terms of scalability and fault tolerance. One of the most important benefits for microservices is the ability to scale easily on demand. It is much more difficult to scale an application when you have thousands of servers running on your network. With microservices, you can scale out individual services instead of scaling out your entire application. This means that you can add more capacity as needed without having to worry about over-provisioning or under-provisioning. The use of CAP, tye, surging libraries makes it easier to work with microservices. These libraries are heavily used and are based on the open-source framework called Capistrano (an automation tool). It allows you to create a deployment script and run it on a remote machine. The CAP pattern allows you to define an interface for your app with which other pieces of code can communicate. The TYE pattern defines an interface for the data that your app needs. Developers tend to use some of the following open source C# Microservice libraries:

java-microservice

11 best Java Microservice

It is very difficult to maintain a large monolithic application. It is also difficult to release new features and bug fixes. But using the Java Microservice libraries like apollo, nacos, armeria can easily solve these problems. The microservice approach has been around for a while now and is being used by many organizations to improve their efficiency, increase the speed of delivery and enhance the quality of their products. Apollo is an open source framework that provides efficient support for building fast and scalable distributed applications with a unified programming model. Apollo uses HTTP as its primary protocol and provides an HTTP client implementation that supports all of the major web browsers (including IE6) as well as other clients such as curl and wget. Apollo also supports streaming data using either Netty or GZIP compression for efficient high-volume data transfers over low bandwidth connections. Nacos is a lightweight library for building reactive asynchronous systems in Java SE 9+. The main purpose of NACOS is to provide a simple yet robust way to create asynchronous applications using Java SE 9+ functional interfaces with low overhead on thread creation overhead and/or blocking calls at the expense of some performance impact due to the need to manage threads yourself. Some of the most popular among developers are:

javascript-microservice

11 best JavaScript Microservice

The use of JavaScript microservice libraries single-spa, moleculer, seneca is not only because it is easy to implement and maintain but also provides a high degree of flexibility. You can easily integrate microservice components into your existing application by using the API. For example, if you want to add a new feature in your application, you can use a JavaScript microservice library like moleculer to develop and test it. They help you to build a large scale application with very little effort, and they allow you to use all of the benefits of the JavaScript language. Moleculer is a JavaScript library that provides an easy and intuitive way to build an API. It is designed to be as simple as possible: it doesn't require an extensive configuration or setup process, but at the same time it allows you to build powerful APIs. Seneca is a lightweight microservice framework that simplifies building, deploying and managing apps. Seneca has recently been rewritten from scratch with Ember (and thus ember-cli), so it supports frontend development with Ember.js apps in addition to backend services with Node and Express (and now both). Some of the most widely used open source JavaScript microservice libraries among developers include:

cpp-microservice

5 best C++ Microservice

On the other hand, if you use C++ microservice libraries like app-mash, edgelessrt, libcluon then this will be perfect for building distributed systems. These libraries have been designed from the ground up to run on multiple machines without any issues and handle distributed systems very well. Libraries like app-mash and edgelessrt are designed with the C++11 standard, so they don’t require you to use any other framework for building your service. Also, these libraries have been tested on Linux and Windows systems and have been used successfully by many developers worldwide. libcluon is also a good choice for building microservices in C++ because it has been tested on both Linux and Windows systems as well as Mac OS X systems. It provides a set of tools that help developers with their daily work: from creating new projects, testing them and deploying them on different platforms (including mobile platforms). App-Mash is a rapid application development framework for easy deployment of web applications. It's built on top of Google App Engine standard library and runs on both Google App Engine and Google Cloud Platform. App-Mash provides a full stack for building robust web applications with all the tools needed to build high performance services: database support, caching, authentication, internationalization support and more. There are several popular open source C++ microservice libraries available for developers:

php-microservice

5 best PHP Microservice

PHP Microservice is an alternative to the traditional web application architecture based on the use of microservices. The main idea behind this architecture is that each service should be completely independent from other services and should have its own state. Microservice libraries like swoft, hyperf, flyimg make it easier to build PHP microservices by providing a common set of tools for developing scalable applications in PHP. Swoft is a microservice framework for developing REST APIs in PHP. It provides a set of tools and libraries that help you build REST API services quickly and easily. Swoft has a variety of features including authentication and authorization, data access strategies, middleware support, as well as support for various HTTP methodologies. Hyperf is a lightweight PHP microservice toolkit built on top of Symfony components. It offers you an easy way to create HTTP services in an efficient, secure way from the ground up. With Hyperf you can easily create: HTTP API, HTTP API with asynchronous calls or callbacks and many other features that will help you build scalable applications quickly. Many developers depend on the following open source PHP Microservice libraries:

ruby-microservice

5 best Ruby Microservice

The Ruby Microservice library is a set of often-used modules that can be used to build more complex applications. The library includes a number of microservices, which are small and simple programs that perform a specific function. The Ruby Microservice library contains many microservices, each designed to solve a different type of problem. Kontena is a project that helps you to manage your full stack CI/CD pipeline, including your Docker registry, Kubernetes clusters, CI/CD pipelines and Docker images management. It has a simple yet powerful UI and integrates with other tools like Jenkins and GitlabCI. Stitches is a library for building complex tasks in an easy way. It makes it possible to use many languages in the same task by providing abstractions over different types of tasks (for example, creating HTML pages or running unit tests) via DSLs written in Python or JavaScript. The library can be used from Ruby as well as from other languages such as Java or Kotlin. Expeditor is a toolkit for managing dockerized applications in AWS environments. You can use it for setting up multi-tier applications on AWS or hybrid cloud infrastructures where you need to run services on-premises and manage them through the cloud provider - AWS EC2 instances, ECS clusters, RDS DB instances etc. Popular open source Ruby Microservice libraries for developers include:

python-microservice

8 best Python Microservice

Python Microservice libraries like falcon, nameko, emissary, etc are becoming popular. These libraries provide you with a lot of benefits like high-performance asynchronous processing, easy integration with NoSQL databases, and REST APIs. These tools help us move away from monolithic systems and towards more scalable ones. They also help us move towards highly available systems that don’t require all of our time spent on infrastructure maintenance and administration. Falcon is an open source Python framework for building scalable web applications. It is backed by Google and supports many different web frameworks such as Flask, Pyramid and Django. Falcon's main purpose is to provide a high-performance asynchronous processing framework for Python web applications. It uses coroutines to provide an event-driven programming model that makes it easy to define complex asynchronous logic and handle multiple concurrent operations within an application. Nameko is a hosted service that lets you create and manage your own DNS records using standard tools. This makes it easy to manage your own DNS records without having to set up servers or worry about security issues. Nameko provides simple collaboration tools so that people can easily create accounts and store data in the cloud without having to worry about securing sensitive information on their own servers. A few of the most popular open source Python Microservice libraries for developers are:

java-microservice

11 best Java Microservice

It is very difficult to maintain a large monolithic application. It is also difficult to release new features and bug fixes. But using the Java Microservice libraries like apollo, nacos, armeria can easily solve these problems. The microservice approach has been around for a while now and is being used by many organizations to improve their efficiency, increase the speed of delivery and enhance the quality of their products. Apollo is an open source framework that provides efficient support for building fast and scalable distributed applications with a unified programming model. Apollo uses HTTP as its primary protocol and provides an HTTP client implementation that supports all of the major web browsers (including IE6) as well as other clients such as curl and wget. Apollo also supports streaming data using either Netty or GZIP compression for efficient high-volume data transfers over low bandwidth connections. Nacos is a lightweight library for building reactive asynchronous systems in Java SE 9+. The main purpose of NACOS is to provide a simple yet robust way to create asynchronous applications using Java SE 9+ functional interfaces with low overhead on thread creation overhead and/or blocking calls at the expense of some performance impact due to the need to manage threads yourself. Some of the most popular among developers are:

cpp-microservice

5 best C++ Microservice

On the other hand, if you use C++ microservice libraries like app-mash, edgelessrt, libcluon then this will be perfect for building distributed systems. These libraries have been designed from the ground up to run on multiple machines without any issues and handle distributed systems very well. Libraries like app-mash and edgelessrt are designed with the C++11 standard, so they don’t require you to use any other framework for building your service. Also, these libraries have been tested on Linux and Windows systems and have been used successfully by many developers worldwide. libcluon is also a good choice for building microservices in C++ because it has been tested on both Linux and Windows systems as well as Mac OS X systems. It provides a set of tools that help developers with their daily work: from creating new projects, testing them and deploying them on different platforms (including mobile platforms). App-Mash is a rapid application development framework for easy deployment of web applications. It's built on top of Google App Engine standard library and runs on both Google App Engine and Google Cloud Platform. App-Mash provides a full stack for building robust web applications with all the tools needed to build high performance services: database support, caching, authentication, internationalization support and more. There are several popular open source C++ microservice libraries available for developers:

ruby-microservice

5 best Ruby Microservice

The Ruby Microservice library is a set of often-used modules that can be used to build more complex applications. The library includes a number of microservices, which are small and simple programs that perform a specific function. The Ruby Microservice library contains many microservices, each designed to solve a different type of problem. Kontena is a project that helps you to manage your full stack CI/CD pipeline, including your Docker registry, Kubernetes clusters, CI/CD pipelines and Docker images management. It has a simple yet powerful UI and integrates with other tools like Jenkins and GitlabCI. Stitches is a library for building complex tasks in an easy way. It makes it possible to use many languages in the same task by providing abstractions over different types of tasks (for example, creating HTML pages or running unit tests) via DSLs written in Python or JavaScript. The library can be used from Ruby as well as from other languages such as Java or Kotlin. Expeditor is a toolkit for managing dockerized applications in AWS environments. You can use it for setting up multi-tier applications on AWS or hybrid cloud infrastructures where you need to run services on-premises and manage them through the cloud provider - AWS EC2 instances, ECS clusters, RDS DB instances etc. Popular open source Ruby Microservice libraries for developers include:

python-microservice

8 best Python Microservice

Python Microservice libraries like falcon, nameko, emissary, etc are becoming popular. These libraries provide you with a lot of benefits like high-performance asynchronous processing, easy integration with NoSQL databases, and REST APIs. These tools help us move away from monolithic systems and towards more scalable ones. They also help us move towards highly available systems that don’t require all of our time spent on infrastructure maintenance and administration. Falcon is an open source Python framework for building scalable web applications. It is backed by Google and supports many different web frameworks such as Flask, Pyramid and Django. Falcon's main purpose is to provide a high-performance asynchronous processing framework for Python web applications. It uses coroutines to provide an event-driven programming model that makes it easy to define complex asynchronous logic and handle multiple concurrent operations within an application. Nameko is a hosted service that lets you create and manage your own DNS records using standard tools. This makes it easy to manage your own DNS records without having to set up servers or worry about security issues. Nameko provides simple collaboration tools so that people can easily create accounts and store data in the cloud without having to worry about securing sensitive information on their own servers. A few of the most popular open source Python Microservice libraries for developers are:

Trending Discussions on Microservice

    Exclude Logs from Datadog Ingestion
    Custom Serilog sink with injection?
    How to manage Google Cloud credentials for local development
    using webclient to call the grapql mutation API in spring boot
    Jdeps Module java.annotation not found
    How to make a Spring Boot application quit on tomcat failure
    Deadlock on insert/select
    Rewrite host and port for outgoing request of a pod in an Istio Mesh
    Checking list of conditions on API data
    Traefik v2 reverse proxy without Docker

QUESTION

Exclude Logs from Datadog Ingestion

Asked 2022-Mar-19 at 22:38

I have a kubernetes cluster that's running datadog and some microservices. Each microservice makes healthchecks every 5 seconds to make sure the service is up and running. I want to exclude these healthcheck logs from being ingested into Datadog.

I think I need to use log_processing_rules and I've tried that but the healthcheck logs are still making it into the logs section of Datadog. My current Deployment looks like this:

1apiVersion: apps/v1
2kind: Deployment
3[ ... SNIP ... ]
4spec:
5  replicas: 2
6  selector:
7    matchLabels:
8      app: my-service
9  template:
10    metadata:
11      labels:
12        app: my-service
13        version: "fac8fb13"
14      annotations:
15        rollme: "IO2ad"
16        tags.datadoghq.com/env: development
17        tags.datadoghq.com/version: "fac8fb13"
18        tags.datadoghq.com/service: my-service
19        tags.datadoghq.com/my-service.logs: |
20          [{
21            "source": my-service,
22            "service": my-service,
23            "log_processing_rules": [
24              {
25                "type": "exclude_at_match",
26                "name": "exclude_healthcheck_logs",
27                "pattern": "\"RequestPath\": \"\/health\""
28              }
29            ]
30          }]
31

and the logs coming out of the kubernetes pod:

1apiVersion: apps/v1
2kind: Deployment
3[ ... SNIP ... ]
4spec:
5  replicas: 2
6  selector:
7    matchLabels:
8      app: my-service
9  template:
10    metadata:
11      labels:
12        app: my-service
13        version: "fac8fb13"
14      annotations:
15        rollme: "IO2ad"
16        tags.datadoghq.com/env: development
17        tags.datadoghq.com/version: "fac8fb13"
18        tags.datadoghq.com/service: my-service
19        tags.datadoghq.com/my-service.logs: |
20          [{
21            "source": my-service,
22            "service": my-service,
23            "log_processing_rules": [
24              {
25                "type": "exclude_at_match",
26                "name": "exclude_healthcheck_logs",
27                "pattern": "\"RequestPath\": \"\/health\""
28              }
29            ]
30          }]
31$ kubectl logs my-service-pod
32
33{
34  "@t": "2022-01-07T19:13:05.3134483Z",
35  "@m": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
36  "@i": "REDACTED",
37  "ElapsedMilliseconds": 7.5992,
38  "StatusCode": 200,
39  "ContentType": "text/plain",
40  "ContentLength": null,
41  "Protocol": "HTTP/1.1",
42  "Method": "GET",
43  "Scheme": "http",
44  "Host": "10.64.0.80:5000",
45  "PathBase": "",
46  "Path": "/health",
47  "QueryString": "",
48  "HostingRequestFinishedLog": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
49  "EventId": {
50    "Id": 2,
51    "Name": "RequestFinished"
52  },
53  "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
54  "RequestId": "REDACTED",
55  "RequestPath": "/health",
56  "ConnectionId": "REDACTED",
57  "dd_service": "my-service",
58  "dd_version": "54aae2b5",
59  "dd_env": "development",
60  "dd_trace_id": "REDACTED",
61  "dd_span_id": "REDACTED"
62}
63

EDIT: Removed 2nd element of the log_processing_rules array above as I've tried with 1 and 2 elements in the rules array.

EDIT2: I've also tried changing log_processing_rules type to INCLUDE at match in an attempt to figure this out:

1apiVersion: apps/v1
2kind: Deployment
3[ ... SNIP ... ]
4spec:
5  replicas: 2
6  selector:
7    matchLabels:
8      app: my-service
9  template:
10    metadata:
11      labels:
12        app: my-service
13        version: "fac8fb13"
14      annotations:
15        rollme: "IO2ad"
16        tags.datadoghq.com/env: development
17        tags.datadoghq.com/version: "fac8fb13"
18        tags.datadoghq.com/service: my-service
19        tags.datadoghq.com/my-service.logs: |
20          [{
21            "source": my-service,
22            "service": my-service,
23            "log_processing_rules": [
24              {
25                "type": "exclude_at_match",
26                "name": "exclude_healthcheck_logs",
27                "pattern": "\"RequestPath\": \"\/health\""
28              }
29            ]
30          }]
31$ kubectl logs my-service-pod
32
33{
34  "@t": "2022-01-07T19:13:05.3134483Z",
35  "@m": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
36  "@i": "REDACTED",
37  "ElapsedMilliseconds": 7.5992,
38  "StatusCode": 200,
39  "ContentType": "text/plain",
40  "ContentLength": null,
41  "Protocol": "HTTP/1.1",
42  "Method": "GET",
43  "Scheme": "http",
44  "Host": "10.64.0.80:5000",
45  "PathBase": "",
46  "Path": "/health",
47  "QueryString": "",
48  "HostingRequestFinishedLog": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
49  "EventId": {
50    "Id": 2,
51    "Name": "RequestFinished"
52  },
53  "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
54  "RequestId": "REDACTED",
55  "RequestPath": "/health",
56  "ConnectionId": "REDACTED",
57  "dd_service": "my-service",
58  "dd_version": "54aae2b5",
59  "dd_env": "development",
60  "dd_trace_id": "REDACTED",
61  "dd_span_id": "REDACTED"
62}
63"log_processing_rules": [
64  {
65    "type": "include_at_match",
66    "name": "testing_include_at_match",
67    "pattern": "somepath"
68  }
69]
70

and I'm still getting the health logs in Datadog (in theory I should not as /health is not part of the matching pattern)

ANSWER

Answered 2022-Jan-12 at 20:28

I think the problem is that you're defining multiple patterns; the docs state, If you want to match one or more patterns you must define them in a single expression.

Try somtething like this and see what happens:

copy icondownload icon

1apiVersion: apps/v1
2kind: Deployment
3[ ... SNIP ... ]
4spec:
5  replicas: 2
6  selector:
7    matchLabels:
8      app: my-service
9  template:
10    metadata:
11      labels:
12        app: my-service
13        version: "fac8fb13"
14      annotations:
15        rollme: "IO2ad"
16        tags.datadoghq.com/env: development
17        tags.datadoghq.com/version: "fac8fb13"
18        tags.datadoghq.com/service: my-service
19        tags.datadoghq.com/my-service.logs: |
20          [{
21            "source": my-service,
22            "service": my-service,
23            "log_processing_rules": [
24              {
25                "type": "exclude_at_match",
26                "name": "exclude_healthcheck_logs",
27                "pattern": "\"RequestPath\": \"\/health\""
28              }
29            ]
30          }]
31$ kubectl logs my-service-pod
32
33{
34  "@t": "2022-01-07T19:13:05.3134483Z",
35  "@m": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
36  "@i": "REDACTED",
37  "ElapsedMilliseconds": 7.5992,
38  "StatusCode": 200,
39  "ContentType": "text/plain",
40  "ContentLength": null,
41  "Protocol": "HTTP/1.1",
42  "Method": "GET",
43  "Scheme": "http",
44  "Host": "10.64.0.80:5000",
45  "PathBase": "",
46  "Path": "/health",
47  "QueryString": "",
48  "HostingRequestFinishedLog": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
49  "EventId": {
50    "Id": 2,
51    "Name": "RequestFinished"
52  },
53  "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
54  "RequestId": "REDACTED",
55  "RequestPath": "/health",
56  "ConnectionId": "REDACTED",
57  "dd_service": "my-service",
58  "dd_version": "54aae2b5",
59  "dd_env": "development",
60  "dd_trace_id": "REDACTED",
61  "dd_span_id": "REDACTED"
62}
63"log_processing_rules": [
64  {
65    "type": "include_at_match",
66    "name": "testing_include_at_match",
67    "pattern": "somepath"
68  }
69]
70"log_processing_rules": [
71  {
72    "type": "exclude_at_match",
73    "name": "exclude_healthcheck_logs",
74    "pattern": "\/health|\"RequestPath\": \"\/health\""
75  }
76

Source https://stackoverflow.com/questions/70687054