datx | opinionated JS/TS data store | State Container library

As I suspected, you are creating a lot of COM objects, but never release them from memory when you are done with them. Eventually, this will create errors from running out of memory resources.

I didn't have time to look into what the functions actually do however, but for the 'cleaning-up' code, I suggest you add all of the below to the end of the functions.

Function Report1:

End the code by quitting Excel end releasing the uses COM objects from memory.

You can use the following batching approach:

• Define the length of a sliding time window that resets if a new file is created within it; keep collecting events while ones arrive within that sliding window.

  • To prevent the collection from growing indefinitely, define a maximum batch size at which a batch is processed, even if further events are pending.

• Once the time window elapses without new events having arrived, process the batch at hand, i.e., the events collected so far, then start a new batch.

• Caveat:

  • The System.IO.FileSystemWatcher class can report duplicate events.

  • The code below eliminates duplicates in a given batch, but not across batches, which would require quite a bit more effort - see the source-code comments.

• Implementation notes:

  • Instead of using an -Action script block passed to Register-ObjectEvent to process the events, they are processed synchronously - with a timeout - in a Wait-Event loop.

  • Wait-Event uses PowerShell's event queue and therefore usually doesn't miss events (although that can happen at the .NET level in high-volume situations); by contrast, the FileSystemWatcher's similar WaitForChanged method does not queue events and only reports a - single - event, if one happens to arrive while the method waits.

It's pretty obvious what's going on here. In order to understand that, you will need to know the difference between the batch_shape and shape arguments in the Input layer.

When you specify the shape argument it actually adds a new dimension (i.e. batch dimension) to the beginning of the shape. Therefore, when you pass a 860x11 as shape, the actual model expects a bx860x11 sized output (where b is the batch size). What you have specified here is the value for the batch_shape argument. So there are two solutions.

The best solution for you would be to change the above to the following. Because this way you're not relying on a fixed batch dimension.

Oh, wow, I couldn't see the wood for the trees, as they say. You're attempting to plot way too many data points (i.e. 100000002, i think that's about 4km length of paper printed at 600dpi), which can be resolved either by sampling:

plot method for your maxima, df3_max, should do the trick:

A plot with a vertical offset sounds like a frequency trail.

Here's one approach that does just adjust the y value.

Frequency Trail in MatPlotLib

The same plot has also been coined a joyplot/ridgeline plot. Seaborn has an implementation that creates a series of plots (FacetGrid), and then adjusts the offset between them for a similar effect.

https://seaborn.pydata.org/examples/kde_joyplot.html

An example using a line plot might look like:

Don't mesh up between mysql and mysqli.

Also you have mixed up query and mysql error function