polars | Fast multi-threaded, hybrid-out-of-core DataFrame library in Rust | Python | Nodejs | GPU library

by pola-rs Rust Version: 0.19.18 License: MIT

X-Ray Key Features Code Snippets(10)Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | polars Summary

polars is a Rust library typically used in Hardware, GPU, Numpy, Pandas applications. polars has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.

Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow.

Support

Quality

Security

License

Reuse

Support

polars has a medium active ecosystem.

It has 17927 star(s) with 967 fork(s). There are 128 watchers for this library.

There were 10 major release(s) in the last 6 months.

There are 808 open issues and 3252 have been closed. On average issues are closed in 26 days. There are 53 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of polars is 0.19.18

Quality

polars has no bugs reported.

Security

polars has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

polars is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

polars releases are available to install and integrate.

Installation instructions, examples and code snippets are available.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of polars

Get all kandi verified functions for this library.

polars Key Features

No Key Features are available at this moment for polars.

polars Examples and Code Snippets

polars - example

Python

Lines of Code : 13

License : Permissive (MIT License)

Copy

import polars as pl
from my_polars_functions import hamming_distance

a = pl.Series("a", ["foo", "bar"])
b = pl.Series("b", ["fooy", "ham"])

dist = hamming_distance(a, b)
expected = pl.Series("", [None, 2], dtype=pl.UInt32)

# run on 2 Series
print(

Filling `null` values of a column with another column

Python

Lines of Code : 39

License : Strong Copyleft (CC BY-SA 4.0)

Copy

import polars as pl

df = pl.DataFrame({'a': [1, None, 3, 4],
                   'b': [10, 20, 30, 40]
                   }).lazy()
print(df.collect())

shape: (4, 2)
┌──────┬─────┐
│ a    ┆ b   │
│ ---  ┆ --- │
│ i

How to create a weighted sum of some columns in a Polars DataFrame?

Python

Lines of Code : 53

License : Strong Copyleft (CC BY-SA 4.0)

Copy

pl.sum([exp1, exp2, etc...])

pl.fold(pl.lit(0), f=lambda c1, c2: c1 + c2, exprs =[expr1, expr2, etc...])

col_names = ["p1", "p2", "p3"]
weights = [7.4, 3.2, -0.13]
df.with_column(
    pl.s

How to use Polars with Plotly without converting to Pandas?

Python

Lines of Code : 22

License : Strong Copyleft (CC BY-SA 4.0)

Copy

import polars as pl
import numpy as np
import plotly.express as px

df = pl.DataFrame(
    {
        "nrs": [1, 2, 3, None, 5],
        "names": ["foo", "ham", "spam", "egg", None],
        "random": np.random.rand(5),
        "groups": ["

Q: How can I append or concatenate two dataframes in python polars?

Python

Lines of Code : 13

License : Strong Copyleft (CC BY-SA 4.0)

Copy

df1 = pl.DataFrame({"a": [1], "b": [2], "c": [3]})
df2 = pl.DataFrame({"a": [4], "b": [5], "c": [6]})


# new memory slab
new_df = pl.concat([df1, df2], rechunk=True)

# append free (no memory copy)
new_df = df1.vstack(df2)

# try to appen

Polars: how to add a column in front?

Python

Lines of Code : 24

License : Strong Copyleft (CC BY-SA 4.0)

Copy

df = pl.DataFrame({
    "a": [1, 2, 3],
    "b": [True, None, False]
})

df.select([
    pl.lit("foo").alias("z"),
    pl.all()
])

shape: (3, 3)
┌─────┬─────┬───────┐
│ z   ┆ a   ┆ b     │
│ --- ┆ --- ┆ ---   │
│ s

Excel equivalent average if on moving window

Python

Lines of Code : 96

License : Strong Copyleft (CC BY-SA 4.0)

Copy

lag_vector = [1, 2, 3]
for lag in lag_vector:
    out = (
        df
        .groupby_rolling(index_column="Date", period=f"{lag}w").agg(
            [
                pl.col('Close Returns').alias('Close Returns list'),
                pl

Python Polars Parse Date from Epoch

Python

Lines of Code : 22

License : Strong Copyleft (CC BY-SA 4.0)

Copy

df = pl.DataFrame({
    "epoch_seconds": [1648457740, 1648457740 + 10]
})

MILLISECONDS_IN_SECOND = 1000;

df.select(
    (pl.col("epoch_seconds") * MILLISECONDS_IN_SECOND).cast(pl.Datetime).dt.and_time_unit("ms").alias("datetime")
)

How to open excel file in Polars dataframe?PythonLines of Code : 5License : Strong Copyleft (CC BY-SA 4.0)

Copy

import polars as pl
import pandas as pd
df = pd.read_excel(...)
df_pl = pl.DataFrame(df)

Python Polars: Read Column as DatetimePythonLines of Code : 39License : Strong Copyleft (CC BY-SA 4.0)

Copy

my_csv = StringIO(
"""
ID,start,last_updt,end
1,2008-10-31, 2020-11-28 12:48:53,12/31/2008
2,2007-10-31, 2021-11-29 01:37:20,12/31/2007
3,2006-10-31, 2021-11-30 23:22:05,12/31/2006
"""
)

pl.read_csv(my_csv, parse_dates=True)

Community Discussions

Trending Discussions on polars

Apply function to all columns of a Polars-DataFrame

Arrow IPC vs Feather

How to define types of columns while loading dataframe in polars?

How to change axes labels to multiples of pi

Drawing lines of constant latitude with cartopy on North Polar Stereo projection

Clip off pcolormesh outside of circular set_boundary in Cartopy

Geopandas - map and locaton plotting

Geopandas world map in Polar Stereographic projection with coloured oceans

Polar Stereographic projection of geopandas world map

Looping a script to produce multiple images

QUESTION

Apply function to all columns of a Polars-DataFrame

Asked 2021-Jun-11 at 09:30

I know how to apply a function to all columns present in a Pandas-DataFrame. However, I have not figured out yet how to achieve this when using a Polars-DataFrame.

I checked the section from the Polars User Guide devoted to this topic, but I have not find the answer. Here I attach a code snippet with my unsuccessful attempts.

...

ANSWER

Answered 2021-Jun-11 at 09:30

You can use the expression syntax to select all columns with pl.col("*") and then map the numpy np.log2(..) function over the columns.

Source https://stackoverflow.com/questions/67834912

QUESTION

Arrow IPC vs Feather

Asked 2021-Jun-09 at 20:52

What is the difference between Arrow IPC and Feather?

The official documentation says:

Version 2 (V2), the default version, which is exactly represented as the Arrow IPC file format on disk. V2 files support storing all Arrow data types as well as compression with LZ4 or ZSTD. V2 was first made available in Apache Arrow 0.17.0.

While vaex, a pandas alternative, has two different functions, one for Arrow IPC and one for Feather. polars, another pandas alternative, indicate that Arrow IPC and Feather are the same.

...

ANSWER

Answered 2021-Jun-09 at 20:18

TL;DR There is no difference between the Arrow IPC file format and Feather V2.

There's some confusion because of the two versions of Feather, and because of the Arrow IPC file format vs the Arrow IPC stream format.

For the two versions of Feather, see the FAQ entry:

What about the “Feather” file format?

The Feather v1 format was a simplified custom container for writing a subset of the Arrow format to disk prior to the development of the Arrow IPC file format. “Feather version 2” is now exactly the Arrow IPC file format and we have retained the “Feather” name and APIs for backwards compatibility.

So IPC == Feather(V2). Some places refer to Feather mean Feather(V1) which is different from the IPC file format. However, that doesn't seem to be the issue here: Polars and Vaex appear to use Feather to mean Feather(V2) (though Vaex slightly misleadingly says "Feather is exactly represented as the Arrow IPC file format on disk, but also support compression").

Vaex exposes both export_arrow and export_feather. This relates to another point of Arrow, as it defines both an IPC stream format and an IPC file format. They differ in that the file format has a magic string (for file identification) and a footer (to support random access reads) (documentation).

export_feather always writes the IPC file format (==FeatherV2), while export_arrow lets you choose between the IPC file format and the IPC stream format. Looking at where export_feather was added I think the confusion might stem from the PyArrow APIs making it obvious how to enable compression with the Feather API methods (which are a user-friendly convenience) but not with the IPC file writer (which is what export_arrow uses). But ultimately, the format being written is the same.

Source https://stackoverflow.com/questions/67910612

QUESTION

How to define types of columns while loading dataframe in polars?

Asked 2021-Apr-17 at 07:19

I'm using polars and I would like to define the type of the columns while loading a dataframe. In pandas, I can use dtype:

...

ANSWER

Answered 2021-Apr-17 at 07:19

The with_schema method expects an Arc type, not a Hashmap.

The following code works:

Source https://stackoverflow.com/questions/67129530

QUESTION

How to change axes labels to multiples of pi

Asked 2020-Oct-27 at 11:29

I am displaying the distribution of data points which I have transformed to polar coordinates, and am displaying the distribution of points using a histogram. How do I change the x axes to be in multiples of pi?

...

ANSWER

Answered 2020-Oct-27 at 11:29

Needed to add:

Source https://stackoverflow.com/questions/64553490

QUESTION

Drawing lines of constant latitude with cartopy on North Polar Stereo projection

Asked 2020-Mar-31 at 16:32

I'd like to draw two lines at 88 and 84 degrees north on a cartopy north polar stereo map, but am stumped as to how to do it.

I've tried with:

...

ANSWER

Answered 2020-Mar-31 at 16:32

This should be available in the next release (0.18). You can test it out if you build/install CartoPy from git master.

Source https://stackoverflow.com/questions/60921747

QUESTION

Clip off pcolormesh outside of circular set_boundary in Cartopy

Asked 2019-Jun-18 at 12:55

I'm using Cartopy for my polar research and would like to clip a circular boundary around my data, which I plot in the NorthPolarStereo() projection. I use set_extent to indicate from what latitude I would like to plot my data and use set_boundary for creating a circular boundary as explained in the gallery. I then use matplotlib.pyplot.pcolormesh to plot the actual data. However, say I use set_extent to define a minimum latitude of 55 degrees, some of my data below 55 degrees is still being plotted outside of my set_boundary. How do I clip off this data?

...

ANSWER

Answered 2019-Jun-18 at 12:55

I don't have cartopy to test it in the same conditions as you, but you can clip a pcolormesh using a Patch object of any shape:

Source https://stackoverflow.com/questions/56649160

QUESTION

Geopandas - map and locaton plotting

Asked 2019-Apr-16 at 14:37

Thanks to the answer to this question I can plot the geopandas world map with continents and oceans coloured in different projections.

Now I would like to add some points, e.g. the cities included in geopandas

...

ANSWER

Answered 2019-Apr-16 at 14:37

The default drawing order for axes is patches, lines, text. This order is determined by the zorder attribute.

Source https://stackoverflow.com/questions/55688279

QUESTION

Geopandas world map in Polar Stereographic projection with coloured oceans

Asked 2019-Apr-12 at 16:13

Adding a further requirement to this question, I also need to have the oceans in blue (or any other colour).

For the 'PlateCarree' projection I can simply do this

...

ANSWER

Answered 2019-Apr-12 at 16:10

You need to plot the map geometries on Cartopy geoaxes, and use cartopy.feature.OCEAN to plot the ocean. Here is the working code that you may try. Read the comments in the code for clarification.

Source https://stackoverflow.com/questions/55651437

QUESTION

Polar Stereographic projection of geopandas world map

Asked 2019-Apr-12 at 12:27

I want to use the geopandas included low resolution world map (see here) as a background for my data. This works fine as long as I use e.g. 'PlateCarree' projection.

If I now want to use a polar stereographic peojection

...

ANSWER

Answered 2019-Apr-12 at 12:27

When plotting with a specific cartopy projection, it is best to actually create the matplotlib figure and axes using cartopy, to make sure it is aware of the projection (in technical terms: to make sure it is a GeoAxes, see https://scitools.org.uk/cartopy/docs/latest/matplotlib/intro.html):

Source https://stackoverflow.com/questions/55646598

QUESTION

Looping a script to produce multiple images

Asked 2019-Feb-02 at 19:14

I have a Netcdf dataset with dimensions [time, height, latitude, longitude] that I've opened with xarray. I've written a code that projects the all the data for a specific timestamp onto a cartopy map and saves the image to my directory. I'd like to create an image for each timestamp, but at the moment the only way I know how to do it is to manually change the timestamp entry and run the code again. Since there are 360 timestamps this would obviously take some time. I know Python is handy for loops, but I'm very unfamiliar with them, so is there a way of embedding this code within a loop so that I can save multiple images in one go?

...

ANSWER

Answered 2019-Feb-02 at 19:14

It seems straightforward to put this in a loop so I am not sure why this is so difficult. Nevertheless, you can try the following. Here I have moved some definitions outside the for loop because you don't need to define them 360 times again and again.

Source https://stackoverflow.com/questions/54496539

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install polars

Install the latest polars version with:.
You can take latest release from crates.io, or if you want to use the latest features / performance improvements point to the master branch of this repo. Required Rust version >=1.58.