ktest | repository contains some infrastructure for running Linux

NULs are used to terminate C strings. From this simple observation, two consequences:

• You cannot possibly pass a NUL inside a single C string (hence the warning from the Python interpreter passing up to you a limitation from the underlying OS).
• If you want to reproduce a given sequence of bytes in an array of C strings, end each string inside that array at the position where a NUL should be.

Thus:

You are very close...

Note that randomName.Next(rows) can return zero. If it does then .Cells[randomName.Next(rows),1] will give you the error you see.

If your sheet has nothing on it then rows will always equal 1 and you have a 50% chance of the random number being zero.

Solution: Set the minimum value to 1 like this: Next(1, rows)

Here's the cause:

Couldn't read data from file

Reasons could include:

1. The file doesn't exist. For example:

I think, the line

Without knowing what's in myTest.py, this is a bit of a guess, but if you're invoking unittest or a test runner when the file is run, then the parameters are interpreted as test modules to run. In other words, unittest is looking for Python test modules called ABC and XYZ. And if ABC.py and XYZ.py don't exist, you'd get exactly the sort of error you're seeing.

If you want to have your own parameters, in addition to unittest's expectations, you can modify your call to main() by passing in arguments directly. For example if you want to use the first to arguments (after the program name) for yourself, and pass the rest to unittest:

How did you deserialize your Kafka record object ? If you use deserializers from spring-kafka API such as StringDeserializer, JsonDeserializer.. I think it's not possible to change record object's package name.

If you write custom Serdes for Kafka record yourself then you can do it in deserialization logic by overriding method readClassDescriptor() of ObjectInputStream. Please find the sample code here.

A couple thoughts:

1) Please post complete code. I don't see where crime_realEstate is defined anywhere. If you leave out the line where you read in your data to that variable, it makes it really hard to reproduce your error, and you're less likely to get help. Also, you should organize all of your import statements so they are at the top of your code. It isn't really a function thing, more of a convention that everyone expects and makes it easier to read.

2) When you reference panel data, are you really talking about a pandas DataFrame? That is sort of the "typical" way to store this kind of stuff for analysis. You may want to get in the habit of referring to data as dataframes so it's clearer to your audience. You should also post the full error traceback so we can see what line of code is bombing exactly.

3) I think you may be misunderstanding PCA, or at least what it is for. PCA (principle component analysis) is a data transformation method, where you are capturing variation in data that is across multiple variables and restating that data as fewer components that capture the same amount (or less, depending on how many components you keep) of variability. Once you run PCA, you won't be able to see which features are contributing to crime, because they will be replaced by totally new components. If it is important to identify the features that are correlated with crime, then PCA is a bad idea.

Please fix items above.

EDIT

I'm not saying PCA is wrong, I'm just saying that the question you asked above ("how do I apply PCA and why is my code bombing"), isn't really the right question. PCA should be used if you think that you have many correlated variables that need to be reduced to a lower level of dimensionality. I wouldn't start there though - see what kind of accuracy you can get without doing that. You've now reformulated to a much broader question of "how do I make a predictive model for this data, preferably using a regression?", which should probably go to https://datascience.stackexchange.com/ instead, but I'll give you a starting point of how I would approach coding that solution.

First - PCA is probably not the ideal starting point because from just looking at the data/columns, your problem isn't dimensionality. You basically have 10 different crimes over 5 years. You also only have 58 different rows...or is that just the sample data? Also, your data is a bit weird - you have the same prices for multiple rows, but the crimes differ. I can't tell if it's just because you're posting sample data. If this is indeed the full dataset, stop your analysis now and get more data/go do something else.

I made some executive decisions about how I would approach the problem. All of these are just for demonstration purposes of how to code regression. I summed crime across all years (you maybe want average? Highest? Change in? Those are all design decisions for you). My metric was Change in Price From 2012-2016, the timeframe you have crime data. I normalized crime counts by type of crime. Didn't scale the target variable.

Here's how I would start:

Melt your dataframe. This way you have separate column for each variable and index by Area_name. :

Intel's documentation saying, "not necessary as they are auto generated by the compiler" is in fact correct. And yet, it's unsatisfying.

But to understand why it is the way it is, you need to look at the history of the AVX512. While none of this information is official, it's strongly implied based on evidence.

The reason the state of the mask intrinsics got into the mess they are now is probably because AVX512 got "rolled out" in multiple phases without sufficient forward planning to the next phase.

Phase 1: Knights Landing

Knights Landing added 512-bit registers which only have 32-bit and 64-bit data granularity. Therefore the mask registers never needed to be wider than 16 bits.

When Intel was designing these first set of AVX512 intrinsics, they went ahead and added intrinsics for almost everything - including the mask registers. This is why the mask intrinsics that do exist are only 16 bits. And they only cover the instructions that exist in Knights Landing. (though I can't explain why KSHIFT is missing)

On Knights Landing, mask operations were fast (2 cycles). But moving data between mask registers and general registers was really slow (5 cycles). So it mattered where the mask operations were being done and it made sense to give the user finer-grained control about moving stuff back-and-forth between mask registers and GPRs.

Phase 2: Skylake Purley

Skylake Purley extends the AVX512 to cover byte-granular lanes. And this increased the width of the mask registers to the full 64 bits. This second round also added KADD and KTEST which didn't exist in the Knights Landing.

These new mask instructions (KADD, KTEST, and 64-bit extensions of existing ones) are the ones that are missing their intrinsic counterparts.

While we don't know exactly why they are missing, there is some strong evidence in support of it:

Compiler/Syntax:

On Knights Landing, the same mask intrinsics were used for both 8-bit and 16-bit masks. There was no way to distinguish between them. By extended them to 32-bit and 64-bit, it made the mess worse. In other words, Intel didn't design the mask intrinsics correctly to begin with. And they decided to drop them completely rather than fix them.

Performance Inconsistencies:

Bit-crossing mask instructions on Skylake Purley are slow. While all bit-wise instructions are single-cycle, KADD, KSHIFT, KUNPACK, etc... are all 4 cycles. But moving between mask and GPR is only 2 cycles.

Because of this, it's often faster to move them into GPRs to do them and move them back. But the programmer is unlikely to know this. So rather than giving the user full control of the mask registers, Intel opted just have the compiler make this decision.

By making the compiler make this decision, it means that the compiler needs to have such logic. The Intel Compiler currently does as it will generate kadd and family in certain (rare) cases. But GCC does not. On GCC, all but the most trivial mask operations will be moved to GPRs and done there instead.

Final Thoughts:

Prior to the release of Skylake Purley, I personally had a lot of AVX512 code written up which includes a lot of AVX512 mask code. These were written with certain performance assumptions (single-cycle latency) that turned out to be false on Skylake Purley.

From my own testing on Skylake X, some of my mask-intrinsic code which relied on bit-crossing operations turned out to be slower than the compiler-generated versions that moved them to GPRs and back. The reason of course is that KADD and KSHIFT was 4 cycles instead of 1.

Of course, I prefer if Intel did provide the intrinsics to give us the control that I want. But it's very easy to go wrong here (in terms of performance) if you don't know what you're doing.

Update:

It's unclear when this happened, but the latest version of the Intel Intrinsics Guide has a new set of mask intrinsics with a new naming convention that covers all the instructions and widths. These new intrinsics supercede the old ones.

So this solves the entire problem. Though the extent of compiler support is still uncertain.

Examples:

• _kadd_mask64()
• _kshiftri_mask32()
• _cvtmask16_u32() supercedes _mm512_mask2int()