nagioscore | Nagios Core | Plugin library

"When Nagios Core attempts to reschedule a host or service check, it will make sure that the next check falls within a valid time range within the defined timeperiod. If it doesn't, Nagios Core will adjust the next check time to coincide with the next "valid" time in the specified timeperiod."

I was actually feeling pretty sure this wouldn't be the case, but maybe this is a bug if you're seeing a different behavior. I would expect the time periods and the check intervals to create a timing issue that would cause many checks to be dropped. Regardless of how things should work and what is/isn't expected behavior, I wouldn't personally configure it like this. Since you say that:

I need a service to be checked three times a day at fixed times.

Here's what I would do, if I were you:

• I would run this check as a cron job, and send in the result of the check as a passive check command to Nagios. This way, you know for sure that the check will always run on time.
• I would then configure a freshness_threshold to ensure that this passive service has actually phoned home recently.
• I would also configure a check_command that prepares for the eventuality of the service not having a fresh result, i.e. something that executes only if no service check has been received -- perhaps a script that re-runs the check and notifies me somehow.

Although there is no standard for format of health checks, you should consider major monitoring tools and their expectations from your protocol. In most cases they react to specific HTTP answer codes. For example Amazon Route 53:

waits for an HTTP status code of 200 or greater and less than 400

Another tool, Consul, has more specific definition:

The status of the service depends on the HTTP response code: any 2xx code is considered passing, a 429 Too Many Requests is a warning, and anything else is a failure.

So you might need to check a few top tools you might integrate later and choose an approach suitable for all of them.

I found this page which describes in some detail how to format the performance data.

This is the expected format:

'label'=value[UOM];[warn];[crit];[min];[max]

Notes:

1. space separated list of label/value pairs
2. label can contain any characters except the equals sign or single quote (')
3. the single quotes for the label are optional. Required if spaces are in the label
4. label length is arbitrary, but ideally the first 19 characters are unique (due to a limitation in RRD). Be aware of a limitation in the amount of data that NRPE returns to Nagios
5. to specify a quote character, use two single quotes
6. warn, crit, min or max may be null (for example, if the threshold is not defined or min and max do not apply). Trailing unfilled semicolons can be dropped
7. min and max are not required if UOM=%
8. value, min and max in class [-0-9.]. Must all be the same UOM. value may be a literal "U" instead, this would indicate that the actual value couldn't be determined

9. warn and crit are in the range format (see the Section called Threshold and ranges). Must be the same UOM

   1. UOM (unit of measurement) is one of:

      • no unit specified - assume a number (int or float) of things (eg, users, processes, load averages)
      • s - seconds (also us, ms)
      • % - percentage
      • B - bytes (also KB, MB, TB)
      • c - a continous counter (such as bytes transmitted on an interface)

It is up to third party programs to convert the Nagios Plugins performance data into graphs.

In my this case the raw data provides a pefect example of the output required.

pending=3;5500;7000 complete=18940;; error=454;;7000

You're using PNP it looks like, which means the actual file that writes the RRD file is in process_perfdata.pl, not Nagios itself. The function that is doing the writing is called data2rrd. What you're asking is essentially a limitation of RRDtool.

If this writing seems expensive to you, you may want to look at setting up RRDcached. RRDcached will log all of the writes that need to occur towards RRD files in a specific directory, and then at a specified interval update them all at the same time.

You can also make sure you have enough buffer cache so that all of your RRD file blocks are essentially updated in memory, and changes to disk are less sporadic. You can do this by ensuring you have adequate memory available to the machine.

Also, you can tune your RRD files (or PNP4Nagios - we'll get to that in a second) to store less information so that their overall size is a bit smaller. The trade-off here is that you'll end up with less historical performance data. This may or may not be an issue for you.

You can tune PNP by looking at /usr/local/nagios/etc/pnp/rra.cfg and /usr/local/nagios/etc/pnp/process_perfdata.cfg. That is where those files are located on my system, your mileage may vary.

Finally: make sure your system is using the latest version of RRDtool. If you installed via yum or apt it may be a bit out of date. I highly recommend compiling from source, as a lot of performance improvements and bug fixes have found their ways in throughout the years.

You can find more information about RRDtool here: http://oss.oetiker.ch/rrdtool/index.en.html

Hope this helps!