ndoutils | NDOUtils - Database Output for Nagios Core | Plugin library

I'm working on a rewrite of a popular add-on module (NDOUtils) for an even more popular application (Nagios). This module adds functionality to Nagios by making the objects/statuses/histories of each object available in a database.

The current available version takes the data from Nagios (via some registered callbacks/function pointers) and sends it over a socket where an additional process listens and queues the data up. Finally, an additional process then pops the data from the queue and builds mysql queries for insertion.

Although it works, and has worked for quite some time, we encounter problems on larger systems (15k+ objects defined in Nagios configurations). We decided to start over and re-write the module to handle the database calls directly (via mysql c api prepared statements).

This works beautifully for the status data. One problem we face is that on startup, we need to get the object definitions into the database. Since the definitions can change each time the process starts, we truncate the appropriate tables and recreate each object. This works fine for most systems...

But for large systems, this process can take upwards of several minutes - and this is a blocking process - several minutes is unacceptable, and this is exacerbated on critical monitoring setups.

So, to get this rewrite underway, I kept things simple. To begin with, I looped over each object definition, and built a simple query and then inserted. Once each of that type of object were inserted, I looped back over the objects for all of the objects that are related (for example, each host definition likely has a contact or contactgroup associated with it. Those relationships need to be identified). This was the easiest to read, but extremely slow on a system with 15k hosts and 25k services. Extremely slow as in 3 minutes.

Of course we can do better than that. I rewrote the major functions (hosts and services) to only need to loop over the object list twice each, and instead of sending an individual query per object or relationship, we build a bulk insert query. The code for this looks something like this: