pickler | PIvotal traCKer Liaison to cucumbER | Functional Testing library

Use the first one and do conso_emi, surf, num_caract = loaded in the second notebook.

The integration tests are designed to be run by Beam's CI/CD infrastructure. They are nose based and require a custom plugin to understand the --test-pipeline-options flag. I wouldn't recommend going this route.

I would follow the quick start guide that Ricco D suggested for the environment. You could use pytest to run the integration test. To use the same --test-pipeline-options flag, you'll need this definition. Otherwise the wordcount example shows how to set up your own command line flags.

Update:

I used this to set up the virtualenv:

It seems you run dask and jupyterlab in docker ?

Maybe you should add some flags like fellowing:

I have taken some time to parse your code.

In the large function, you use s3fs to interact with your cloud storage, and this works well with xarray.

However, in your main code, you use boto3 to list and open S3 files. These files retain a reference to the client object, which maintains a connection pool. That is the thing that cannot be pickled.

s3fs is designed to work with Dask, and ensures the picklebility of the filesystem instances and OpenFile objects. Since you already use it in one part, I would recommend using s3fs throughout (but I am, of course biased, since I am the main author).

Alternatively, you could pass just the file names (as strings), and not open anything until within the worker function. This would be "best practice" - you should load data in worker tasks, rather than loading in the client and passing the data.

That error is usually caused by using the save_main_session=True option. See Handle nameerrors when launching Dataflow jobs with Apache Beam notebooks for a discussion on other ways of making sure the workers have the right code available at runtime.

Solved: The error gave me one clue. Byte class object which I tried to pickle is a FormatField and has nothing to do with the Struct class. Only after few hours of thinking about it, it occured to me that Struct refers to struct.Struct and not construct.Struct. After getting rid of it, it serializes properly.

Empty construct.Struct class object serializes without issues.

Offending code:

Yes, seems like a simple solution as

The short answer to your question is to use pickle.Unpickler() to load your pickles.

Alternatively, don't use pickle.Pickler(). Instead write each pickle with pickle.dump() and read back with pickle.load() or pickle.Unpickler().

Either one of those should cure your problem.

I can confirm that the same problem that you describe exists in Python 3.9.1 for both protocol versions 4 and 5.

BTW: notice that {"a",3} in your last example is a set, not a dict as you thought. Nevertheless the same error will occur.

The problem is that Pickler uses a memo to cache data that it has pickled. It uses this to economise on the size of the resulting file by avoiding storing the same data more than once. The memo is shared between all pickles written with the Pickler.

The Unpickler uses the memo to reconstruct the pickled objects that share cached data. However, pickle.load() does not use the memo and can therefore fail to find the values that the Pickler memoized when it dumped the individual pickles.

Here is some code to demonstrate:

Instead of using save_main_session, unpack the extract words outside ReadWordsFromText composite transform. Here is the example: