in-seconds | time calculator for music applications | Apps library

I ran several streaming spark jobs and batch spark jobs in the same EMR cluster. Recently, one batch spark job is programmed wrong, which consumed a lot of memory. It causes the master node not response and all other spark jobs stuck, which means the whole EMR cluster is basically down.

Are there some way that we can restrict the maximum memory that a spark job can consume? If the spark job consumes too much memory, it can be failed. However, we do not hope the whole EMR cluster is down.

The spark jobs are running in the client mode with spark submit cmd as below.

In my project I'm using Jhipster Spring Boot and I would like to start 2 instances of one microservise at the same time, but on different instances of a database (MongoDB).

In this microservice I have classes, services, rests that are used for collections A, B C,.. for which now I would like to have also history collections A_history, B_history, C_history (that are structured exactly the same like A, B, C) stored in separated instance of a database. It makes no sense to me to create "really separated" microservice since I would have to copy all logic from the first one and end up with doubled code that is very hard to maintain. So, the idea is to have 2 instances of the same microservice, one for A, B, C collections stored in "MicroserviceDB" and second for A_history, B_history, C_history collections stored in "HistoryDB".

I've tried with creating 2 profiles, but when I start from a command line History microservice, it is started ok, but if I also try to start "original" microservice at the same time, it is started but immediately history service becomes "original" microservice. Like they cannot work at the same time.

Is this concept even possible in microservice architecture? Does anyone have an idea how to make this to work, or have some other solution for my problem?

Thanks.

application.yml

I'm currently in the process of adding some metrics to an existing pipeline that runs on Google Dataproc via the Spark Runner and I'm trying to determine how to access these metrics and eventually expose them to Stackdriver (to be used downstream in Grafana dashboards).

The metrics themselves are fairly simple (a series of counters) and are defined as such (and accessed in DoFns throughout the pipeline):

Here's my problem. I'm using @Scheduled on top of some methods to do some regular tasks in my spring-boot application.

For configuring the schedule I'm using the fixedDelayString & initialDelayString arguments to @Scheduled as follows:

My Payara REST API is working fine without DB Connection, and also Postgres localhost connection works fine through DriverManager i.e.:

I'm developing an application with microservice architecture using jhipster. I can run my services in dev mode even though i get this warning but when I run it on kubernetes cluster after i get this warning pod restarts itself over and over on loop. I have 4 microservies and a gateway. All the same. Thank you in advance.

This is the warning:

I have a microservice arhitecture and each microservice has it's own mysql database. The application works fine in an IDE but i have problems deploying it because I can't bind my mysql db to game microservice. I'm getting Communication link failure. Stack trace is below.

docker-compose.yml

I'm looking to connect a Kinesis stream to a Lambda function via event source mapping, want to set the parallelization-factor value to any value between 1- 10 as suggested in the documention:

https://docs.aws.amazon.com/cli/latest/reference/lambda/create-event-source-mapping.html

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html#cfn-lambda-eventsourcemapping-startingposition

and an example at https://aws.amazon.com/blogs/compute/new-aws-lambda-scaling-controls-for-kinesis-and-dynamodb-event-sources/

The following command results in an error:

I'm trying to change the maximum-event-age setting for Lambdas using a bash script. Serverless does not currently appear to support this setting, so I'm planning to do it as a bash script after a deploy from GitHub.

Approach:

I'm considering querying aws for the Lambdas in a specific CloudFormation stack. I'm guessing that when a repo is deployed, a new CF stack is created. Then, I want to iterate over the functions and use the put-function-event-invoke-config to change the maximum-event-age setting on each lambda.

Problem:

The put-function-event-invoke-config seems to require a function name. When querying for CF stacks, I'm getting the lambda ARNs instead. I could possibly do some string manipulation to get just the lambda name, but it seems like a messy way to do it.

Am I on the right track with this? Is there a better way?

Edit:

The lambdas already exist and have been deployed. What I think I need to do is run some kind of script that is able to go through the list of lambdas that have been deployed from a single repository (there are multiple repos being deployed to the same environment) and change the maximum-event-age setting that has a default of 6 hours.

Here's an example output when I use the CLI to query CFN with aws cloudformation describe-stacks :