sparser | Sparser : Raw Filtering for Faster Analytics over Raw Data | Runtime Evironment library

My intuition says that a High Dynamic Range image would provide more stable features and edges for various image segmentation and other low level vision algorithms to work with - but then it could go the other way with the larger number of bits leading to sparser features as well as the extra cost involved in generating HDR if it needs to be derived using exposure fusion or such instead of from hardware.

Can anyone point out any research on the topic, ideally it would be good to find out if there has been a comparison study for various machine vision techniques using Standard and High dynamic range images.

I need to create large (see more on this below) random graphs to compare performance of Dijkstra, Bellman-Ford, and Floyd's algorithms on shortest path graph traversal. I'm storing the adjacencies in an array. So far, I generated random weights between vertices, and filled the main diagonal with 0's. I also have symmetry about the main diagonal (I'm assuming the graphs are undirected but not necessarily completely connected).

The random values are in the range 0 - 24 ish, generated using rand() % 25. The problem is that I'd like the graphs to be sparser (i.e. have less edges). Is there a way to generate random numbers within a range and have about 1/3 to 1/2 of the generated numbers be a specific value? Note that the random distribution isn't very important for what I'm doing...

Another question: how large of a graph should I test to see performance differences? 10 vertices? 100? 1000? 10000000?

Given a sparse matrix A and a vector b, I would like to obtain a solution x to the equation A * x = b as well as the kernel of A.

One possibility is to convert A to a dense representation.

Our cosmos db aggregate query seems slow and costs a lot of RUs. Here are the details (plus see screenshot below): 2.4s and 3222RUs to count a result set of 414k records. Also this for just one count. Normally we would want to do a sum on many fields at once (possible only within a single partition), but performance for that is much worse.

There are 2 million records in this collection. We are using Cosmos DB w/SQL API. This particular collection is partitioned by country_code and there are 414,732 records in France ("FR") and the remainder in US. Document size is averages 917 bytes and maybe min is 800 bytes, max 1300 bytes.

Note that we have also tried a much sparser partitioning key like device_id (of which there are 2 million, 1 doc per device here) which has worse results for this query. The c.calcuated.flag1 field just represents a "state" that we want to keep a count of (we actually have 8 states that I'd like to summarize on).

The indexing on this collection is the default, which uses "consistent" index mode, and indexes all fields (and includes range indexes for Number and String). RU setting is at 20,000, and there is no other activity on the DB.

So let me know your thoughts on this. Can Cosmos DB be used reasonably to get a few sums or counts on fields without ramping up our RU charges and taking a long time? While 2.4s is not awful, we really need sub-second queries for this kind of thing. Our application (IoT based), often needs individual documents, but also sometimes needs these kinds of counts across all documents in a country.

Is there a way to improve performance?

I am building an example to show, graphically, how the least square method works. I am applying a numerical approach where I feed R a number of combinations of possible values of intercept (a) and slope (b), then compute the sum of squares (SSE) for all possible combinations. The combinations of a and b with associated the lowest SSE should be the best one, but somehow my estimates of a are always off mark compared to the real value computed by lm(). On top of that, My estimate of a is sensitive to the range of possible values of a given to R - the broader the range, the more the estimate of a is off.

Here is my example. I'm using the dataset "longley", built in R:

I am using scipy to generate a sparse finite difference matrix, constructing it initially from block matrices and then editing the diagonal to account for boundary conditions. The resulting sparse matrix is of the BSR type. I have found that if I convert the matrix to a dense matrix and then back to a sparse matrix using the scipy.sparse.BSR_matrix function, I am left with a sparser matrix than before. Here is the code I use to generate the matrix:

Edited to add code:

I am trying to replicate some work from a colleague that uses SAS. We're having an issue with the import in SAS which converts text (which matches boolean) to numeric.

The purpose of this work is to identify particular records to pass on, so we need the values to be preserved as originally imported (something I think R will be able to do). Right now we're fixing the issue manually because it's a small number of records but that may not always be true.

Where I'm hitting a snag is that I need to replicate their matrix array in R. There are multiple conditions that should be flagged with a 1 if they meet the condition, as follows: SAS Code

I need to be able to evaluate if there is one of 34 potential strings (or partial strings (in SAS, the colon shortens a comparison value to the same length as the evaluation value and compares them) in one of 12 columns (e.g. :Q16 means the string only need start with Q16). Additionally, any one of the 12 could have a value through it does get sparser in later fields.

I am trying to find the most efficient and compact approach, if possible.

I'm still somewhat new at R for more complex problems so I am stymied. I've tried a few approaches with grep and grepl but none have born any fruit. When I tried regex, I tried using each string individually in ifelse and then I also tried one larger string with the "|" operator but no luck either. I also tried base (apply) and dplyr approaches.

Any help is appreciated.

The structure of the data is: Example Table

Code for Example Data:

My small AWS EC2 instance runs a two python scripts, one to receive JSON messages as a web-socket(~2msg/ms) and write to csv file, and one to compress and upload the csvs. After testing, the data(~2.4gb/day) recorded by the EC2 instance is sparser than if recorded on my own computer(~5GB). Monitoring shows the EC2 instance consumed all CPU credits and is operating on baseline power. My question is, does the instance drop messages because it cannot write them fast enough?

Thank you to anyone that can provide any insight!

I'm working on a project involving solving large underdetermined systems of equations.

My current algorithm calculates SVD (numpy.linalg.svd) of a matrix representing the given system, then uses its results to calculate the Moore-Penrose pseudoinverse and the right nullspace of the matrix. I use the nullspace to find all variables with unique solutions, and the pseudo-inverse to find out it's value.

However, the MPP (Moore Penrose pseudo-inverse) is quite dense and is a bit too large for my server to handle.

I found the following paper which details a sparser pseudoinverse that maintains most of the essential properties of the MPP. This is obviously of much interest to me, but I simply don't have the math background to understand how he's calculating the pseudoinverse. Is it possible to calculate it with SVD? If not, what's the best way to go about it?

These are the lines of the paper which I think are probably relevant but I'm not antiquated enough to understand

• spinv(A) = arg min ||B|| subject to BA = In where ||B|| denotes the entrywise l1 norm of B

• This is in general a non-tractable problem, so we use the standard linear relaxation with the l1 norm

• sspinv(A) = ητ {[spinv(A)]}, with ητ (u) = u1|u|≥τ

Find my code and more details on the actual implementation here