pickling | Fast , customizable , boilerplate-free pickling support | Serialization library

Save it as one.py

Call data.decode() or the equivalent str(data, encoding='utf-8') to convert the bytes to a valid base64-encoded string:

Yes, a try/except is the best way to go about this.

Per the docs, pickle is capable of recursively pickling objects, that is to say, if you have a list of objects that are pickleable, it will pickle all objects inside of that list if you attempt to pickle that list. This means that you cannot feasibly test to see if an object is pickleable without pickling it. Because of that, your structure of:

Your google-cloud-bigtable version is too high.

There is some movement in updating apache-beam dependencies here

They have the same issue. Can you roll back your bigtable version to something before 2? If you run this:

The issue rooted in the model version used for the xgboost framework. from 1.3.0 on the default output changed from pickle to json and the sagemaker documentation does not seem to have been updated accordingly. So if you want to read the model via

I'm not sure if this will help, but your reStructuredText has random indentation and mismatched heading underline lengths. White space matters in reStructuredText. Try cleaning it up, like so:

Maybe the original function and the wrapped one not having the same id is the problem?

In a way, yes. Before the function is sent to the target process it's pickled, which fails in your case, because the func object in the decorator's scope is different from the fact object in your main module, after rebinding by the decorator. Look at this and this question for some background.

Based on these answers I created an example on how what you want can be achieved. The trick is to create a picklable "runner" function, that the target process can use to look up your orginal function from some sort of registry (e.g. a dict...) and run it. This is of course just an example. You will probably not want to create your ProcessPoolExecutor in the decorator.

Usually, in any of programming language, you don't save the object but instead, you save the attributes of an object into a file. Then you should make a function to read the file and fill the value into your object. You can make your object into an installable package and import the package as an object into your program.

In its default backend, celery uses a BLOB column to store its result in the celery_taskmeta table which is limited to 64K in mysql. Somewhere in your celery logs you're probably also seeing a truncation warning from mysql when the result is being written to that table.

The celery result isn't really intended for passing around large files, but more just to hold some minimal details about the result of your task.

You don't give many details about your use case, but generally writing such a large binary blob to your database is a smell, or at least a headache waiting to happen some day.

A decent workaround would be to write the file to the file system or upload it to your favorite elastic storage area then return the file name in the task's result. The concern there would be to make sure your worker node has access to the same file system as the node that needs the result.

You might also be able to get away with altering the celery_taskmeta table to allow a MEDIUMBLOB or LONGBLOB to be stored in the table, but my instincts are that you'll eventually wish you weren't storing the files in the RDBMS anyway. You'll have to make sure that alteration is made every time you deploy your application from scratch.