rrdtool | SVN-import of trunk from svn.oetiker.ch | Version Control System library

Step 1: Download this stable release of uWSGI

Step 2: Extract the tar file inside the site-packages folder of the virtual environment.

For example the extracted path to uwsgi should be:

Found solution on github:

https://github.com/unbit/uwsgi/issues/2361

LDFLAGS=-L/opt/homebrew/Cellar/gettext/0.21/lib pip install --no-cache-dir "uWSGI==2.0.20"

According to the doc, python3 requires return type of bytes.

It's hard to give a general answer as requirements are different for everyone but I can say in my experience that the normal practice is to have a single date-partitioned HDB as this can accommodate the widest range of historical datasets. In terms of increasing granularity of aggregation:

1. Full tick data - works best as date-partitioned with `p# on sym
2. Minutely aggregated data - still works well as date-partitioned with `p# on either sym or minute, `g# on either minute or sym
3. Hourly aggregated data - could be either date-partitioned or splayed depending on volume. Again you can have some combination of attributes on the sym and/or the aggregated time unit (in this case hour)
4. Weekly aggregated data - given how much this would compress the data you're now likely looking at a splayed table in this date-partitioned database. Use attributes as above.
5. Monthly/Yearly aggregated data - certainly could be splayed and possibly even flat given how small these tables would be. Attributes almost unnecessary in the flat case.

Maintaining many different HDBs with different partition styles would seem like overkill to me. But again it all depends on the situation and the volumes of data involved and in the expected usage pattern of the data.

This is due to data consolidation being performed prior to the VDEF calculation.

Although your rrdtool graph arguments specify a step of 300s, this is less width than a pixel of the graph, and so the data series are further averaged before you get to the VDEF. All the CDEF and VDEF functions will always work with a time series of one cdp per pixel. From the RRDTool manual:

Note: a step smaller than one pixel will silently be ignored.

This means that, while you can decrease the resolution of the data, you cannot increase it. Sadly, to get an accurate 95th Percentile, you need higher-resolution data.

So, if you omit the --step 300 in a narrow graph, what will happen is:

• You ask for a 1-month time window
• RRDTool calculates 1 pixel is about 1 hour
• DS retrieves an Average time series from the 1hour RRA, one cdp per pixel (IE hour)
• VDEF then consolidates this to a 95th percentile
• The 95th percentile calculation is inaccurate

With the --step 300 it is slightly different process, but the same result:

• You ask for a 1-month time window, with step 300
• RRDTool calculates 1 pixel is about 1 hour
• RRDTool DS retrieves a month's worth of data from the 300s RRA
• RRDTool further consolidates this data down to 1cdp per pixel (IE per hour) using Average
• VDEF then consolidates this to a 95th percentile
• The 95th percentile calculation is inaccurate

So, you can see the final outcome is the same - its just where the 300s -> 1h consolidation happens, either in the RRA or at graph time.

When using a wide graph, the time per pixel becomes smaller, and RRDTool then no longer needs to perform its additional consolidation of the data, resulting in a more accurate calculation:

• You ask for a 1-month time window
• RRDTool calculates 1 pixel is about 5 minutes
• RRDTool DS retrieves a month's worth of data from the 300s RRA
• No further consolidation is required
• VDEF then consolidates this to a 95th percentile
• The 95th percentile calculation is accurate!

When you retrieve the raw data using rrdtool fetch1 then this extra consolodation doesn't happen, so you get:

• You ask for a 1-month time window with step 300
• RRDTool DS retrieves a month's worth of data from the 300s RRA
• These data are output
• Your spreadsheet then calculates a 95th percentile
• The 95th percentile calculation is correct (well, as close as you can be with a 5min interval)

Your next question will likely be, how do I stop this from happening? The unfortunate answer is that you cannot. RRDTool does not have a Percentile type CF, and so the correct calculations cannot be performed in the RRA (this would be the only real solution).

The Routers2 frontend for MRTG calculated 95th Percentiles for the graphs, and the way it does it is to perform a high-resolution fetch to get the raw data and calculates the value internally before passing this in a HRULE when making the graph. In other words, it doesn't use a VDEF at all, due to this problem you are experiencing.

This is because of implicit averaging in your graph.

First, lets leave aside the question of why your RRA 0, 1 and 2 are all of type AVERAGE with 1cdp=1pdp. Your RRA 0 and 1 are totally superfluous. You may want to review how you create your RRD file.

When you do a rrdtool graph then you're going to be using the 1cdp=1pdp RRA, which is in effect the raw data. However, once the data width gets larger than 1cdp per pixel, RRDTool will implicitly consolodate the data before displaying.

If you're showing a weekly graph, then you'll end up with 1 pixel per cdp (well actually 672 cdp for 600 pixels, but its close enough), and so you'll be displaying the stored value in the RRA (in this case, 4.0). However, when you do it for a month, you'll have 4 times as many samples but the same number of pixels. Thus, your displayed pixels will in fact be the average of 4 data points (well, 4.48cdp to be precise). This means that, if the data are irregular, it will smooth out the data and you'll get lower peaks, which is what you're seeing (a value of 2.1 in your example).

When you do a rrdtool fetch then you're pulling the data straight from the RRA without consolodation (unless you explicitly request it). So, in this case, you see the 4.0 again.

What to do about this depends on what you want to see. Since you have a MAX RRA at the higher resolution, you have a few options -

• Graph two lines for your monthly graph - an AVERAGE, and a MAX. You may also want to add a MIN RRA and show that line, depending on how spiky tha data are.
• Use your AVERAGE RRA, but explicitly tell RRDTool (in the DS definition) to use MAX as the secondary consolodation method. This is not so good as it gives a misleadingly high value but it is more what you're expecting to see
• Keep it this way, and add some text to explain the data have been averages over the graph pixel time interval
• Make the graph 4x wider so that the number of pixels again matches the number of cdp and additional consolodation is not required

Try using this: