pyreadstat | Python package to read sas | Data Manipulation library

Dates, times and datetimes are always stored in SPSS as a number and then you add a format for displaying. SPSS continuously adds new formats while removes others. New formats have to be added manually to the pyreadstat code, while old formats stay in the code for backward compatibility. So the problem is you have found a new Date/datetime/time format that is not registered in pyreadstat.

Another workaround would have been to open the file in SPSS and store it as a date/datetime/time, but with a different format pyreadstat would recognise, for example DATE11, DATETIME20 etc (the current list that pyreadstat accepts is [https://github.com/Roche/pyreadstat/blob/master/pyreadstat/_readstat_parser.pyx#L52-L54])

The best when this is found is to submit a github issue describing the new format found for it to be added. I just added a few I found in the most recent SPSS documentation, and hopefully your problem should be solved in the next release (already available on dev). If not, please submit an issue with a reproducible example.

The numbers SPSS uses to store the dates are not unix time, but either the number of seconds (in the case of datetimes or time) or days (in the case of dates) since 1582-10-14 (the start of the Gregorian Calendar. So you would need something like this to calculate it manually:

Not perfect but it's a first step, you can use meta.__dict__

It is not the encoding, it is coming from a change in Readstat (the C library behind pyreadstat) where they admit files with versions being only numeric, while some files have letters on it ( V for viya or visual). Versions of pyreadstat prior to 0.3 would work.

The issue is tracked here in pyreadstat and here.

Once the issue is repaired in Readstat, it will be fixed in Pyreadstat.

If you only need type, then it is easy: in pyreadstat if $ then it is charachter, if not, it is numeric.

What you are seeing in pyreadstat is what you have in the format column of SAS without the variable width (which is stored separately in pyreadstat in meta.variable_display_width). You will observe in your screenshot that all character variables have a format that starts with $ , the number that comes next is the variable width.

SAS has only two types: charachter and number, therefore if not a character it is a number. The format tells SAS how to display the variable. For characters it is just display the charachter ($) with cerain width, as there are no more alternatives. Numbers can be displayed in different ways, things like BEST, but also as DATE if they represent the number of days since Jan 1st 1960, as DATETIME if they represent the number of seconds since Jan 1st 1960, etc.

In case formats are missing, you can check if the data in a column is a string, in which case the type in SAS was character. Anything else was numeric:

I would do it like this:

Let's try reindex:

For this specific case I would suggest a different approach: you can give an argument "encoding" to pyreadstat.read_sav to manually set the encoding. If you don't know which one it is, what you can do is iterate over the list of encodings here: https://gist.github.com/hakre/4188459 to find out which one makes sense. For example:

Found an answer on the stats section: https://stats.stackexchange.com/questions/243207/what-is-the-proper-usage-of-scale-pos-weight-in-xgboost-for-imbalanced-datasets

scale_pos_weight seems to be a parameter that you can adjust to deal with imbalances in classes like this. Mine was set to the default, 1, which means that negative (0) and positive (1) cases are assumed to be showing up evenly. If I change this to 4, which is my ratio of negatives to positives, I start seeing cases predicted into 1.

My accuracy score goes down, but this makes sense: you get a higher % accuracy with this data by predicting everyone to be 0 since the vast majority of cases are 0, but I want to run this model not for accuracy but for information on the importances/contributions of each predictor, so I want differing predictions.

One answer in the link also suggested being more conservative by setting scale_pos_weight to the sqrt of the ratio, which would be 2, in this case. I got a higher accuracy with 2 than 4, so that's what I'm going with, and I plan to look into this parameter in future classification models.

For a multi-class model, it looks like you're better off adjusting the case-level weights to bring your classes to even representation, as outlined here: https://datascience.stackexchange.com/questions/16342/unbalanced-multiclass-data-with-xgboost

The read_spss function can only read from a local file path:

path: pathstr or Path - File path.

Compare that with the read_csv function:

filepath_or_bufferstr: str, path object or file-like object - Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected.