ddf | DDF Distributed Data Framework - an open source | Authorization library

One potential problem is line:

Yes, because of the way eval works, essentially, since you are implicitly passing a different object for locals and globals (since it just gets globals() and locals()), the expression is being evaluated as if it were in a class definition, which does not create an enclosing scope. The behavior is described in the docs:

If the locals dictionary is omitted it defaults to the globals dictionary. If both dictionaries are omitted, the expression is executed with the globals and locals in the environment where eval() is called.

and if to keep reading in the exec docs:

Remember that at the module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

And class scopes don't create enclosing scopes, which is why you can't call methods without using self.

Here is another way to reproduce:

You can use tidyr::complete:

The error you get is the following

A few suggestions:

• if your function is simple, then it is not necessary to pass the series as an argument, so something like ddf.apply(myfunc, axis=1) should work. If the function takes multiple arguments, then content of the function should specify how to handle multiple columns.

• turns out json doesn't like numpy dtypes, so before dumping the value needs to be converted using int.

• if the dataframe is saved to csv, then there is no neeed to .compute it before, as it will involve doing same work twice.

• if myfunc does not depend on the neighbouring rows, one could also use .map_partitions.

Does this work -

The result is not strange, it's the right behavior: apply returns a value for the group, here 1 and 2 which becomes the index of the aggregation:

Look at your b array

The raw file s3://nyc-tlc/trip data/yellow_tripdata_2010-02.csv contains an error (one too many commas). This is the offending line (middle) and its neighbours: