cloudwatch_exporter | Metrics exporter for Amazon AWS CloudWatch | AWS library

I had the same problem, the problem is the permissions on /data/prometheus should be set to prometheus user and group.

so the sollution is: sudo chown -R prometheus:prometheus /data/prometheus/

actually in your case that path is /app/prometheus/data

I don't have any metrics with such low resolution, so I can't actually test this. But I suppose the problem is that the resulting step when zooming out is longer than 5 minutes (Prometheus' default "lookback delta", configurable via the --query.lookback-delta command-line flag).

What this lookback delta means is that if you query the value of a time series at a point in time (either via an instant query, or via a range query that will retrieve the values of said time series at distinct points in time, separated by a fixed step time) you'll only get back a value if there is a sample less than 5 minutes before. So with a step much larger than 5 minutes and samples much farther apart than 5 minutes, it's less likely for your query to hit timestamps less than 5 minutes after a sample and more likely to hit timestamps where there is no sample within the preceding 5 minutes.

So what you need to do is extend this 5 minute time range. Either via the command line flag above (not recommended, as it may break other assumptions, for time series with more reasonable scrape intervals); or by using one of the _over_time functions. In combination with Grafana's $__interval variable, something like this should work: