mesos | Apache Mesos is a cluster manager

• pyspark 2.1.0
• python 3.5.2

I have a join with multiple conditions:

I am using Mesos 1.3.1 and Chronos in my local. I currently have 100 jobs scheduled every 30 minutes for testing.

Sometimes the tasks get stuck in RUNNING status forever until I restart the Mesos agent that the task is stuck. No agent restarted during this time.

I have tried to KILL the task but its status never gets updated to KILLED while the logs in Chronos say that successfully received the request. I have checked in Chronos that it did update the task as successful and end time is also correct but duration is ongoing and the task is still in RUNNING state.

Also the executor container is running forever for the task that are stuck. I have the executor container that will sleep for 20 seconds and set the offer_timeout to 30 seconds and executor_registration_timeout to 2 minutes.

I have also included Mesos reconciliation every 10 minutes but it updates the task as RUNNING every time.

I have also tried to force the task status to update again as FINISHED before the reconciliation but still not getting updated as FINISHED. It seems like the Mesos leader is not picking up the correct status for the stuck task.

I have tried to run with different task resource allocations (cpu: 0.5,0.75,1...) but does not solve the issue. I changed the number of jobs to 70 for every 30 minute but still happening. This issue is seen once per day which is very random and can happen to any job.

How can I remove this stuck task from the active tasks without restarting the Mesos agent? Is there a way to prevent this issue from happening?

I have two Django websites that create a Spark Session to a Cluster which is running on Mesos.

The problem is that whatever Django starts first will create a framework and take 100% the resources permanently, it grabs them and doesn't let them go even if idle.

I am lost on how to make the two frameworks use only the neede resources and have them concurrently access the Spark cluster.

Looked into spark schedulres, dynamic resources for spark and mesos but nothing seems to work. Is it even possible or should I change the approach?

I try to run my code written in main2.py

I have a flask API that using pyspark starts spark and sends the job to a mesos cluster.

The executor fails because it's taking part of the route where spark-class is located in the Flask API.

Logs:

In a Mesos ecosystem(master + scheduler + slave), with the master executing tasks on the slaves, is there a configuration that allows modifying number of tasks executed on each slave?

Say for example, currently a mesos master runs 4 tasks on one of the slaves(each task is using 1 cpu). Now, we have 4 slaves(4 cores each) and except for this one slave the other three are not being used.
So, instead of this execution scenario, I'd rather prefer the master running 1 task on each of the 4 slaves.

I found this stackoverflow question and these configurations relevant to this case, but still not clear on how to use the --isolation=VALUE or --resources=VALUE configuration here.

Thanks for the help!

Can't access Mesos UI on the master:5050.

Running a small cluster 1 master-3 slaves on a Linux dist.

Master seems to be picked OK:

I'm trying to modify prometheus mesos exporter to expose framework states: https://github.com/mesos/mesos_exporter/pull/97/files

A bit about mesos exporter - it collects data from both mesos /metrics/snapshot endpoint, and /state endpoint. The issue with the latter, both with the changes in my PR and with existing metrics reported on slaves, is that metrics created lasts for ever (until exporter is restarted). So if for example a framework was completed, the metrics reported for this framework will be stale (e.g. it will still show the framework is using CPU).

So I'm trying to figure out how I can clear those stale metrics. If I could just clear the entire mesosStateCollector each time before collect is done it would be awesome. There is a delete method for the different p8s vectors (e.g. GaugeVec), but in order to delete a metric, I need to not only the label name, but also the label value for the relevant metric.

My question is similar to :

Standalone Spark cluster on Mesos accessing HDFS data in a different Hadoop cluster

While the question above is about using spark to process data from a different hadoop cluster, I would also like to know how the spark processes data from azure blob storage container.

From the azure documentation (https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-storage), the following code is used to load the data directly into a dataframe: