DeepCopy | Create deep copies of your objects | Game Engine library

It looks like mypy doesn't understand the abstractclassmethod decorator. That decorator has been deprecated since Python 3.3, as the abstractmethod and classmethod decorators were updated to play nice together. I think your code will work properly if you do:

Once your raw_data dictionary has been created, I don't see where it is ever modified (after all, that is the point of using deepcopy on it). So while deep-copying a mutable object is not thread safe, this particular object is not undergoing mutation at any time. So I don't see why there would be an issue. But, you could always do the deepcopy under control of a lock if you were not confident.

If you are doing this with multithreading, then using a threading.Lock is probably not going to cost you in performance since the deepcopy operation is all CPU and you cannot achieve any deepcopy parallelism anyway because your thread is already locking on the Global Interpreter Lock (GIL) for that function (it is primarily Python bytecode). This additional locking just prevents giving up your time slice while in the middle of a deepcopy operation to another thread that might begin a deepcopy operation (but again, I still don't think that is an issue). But if you are using multithreading, then what performance increase will you be getting from doing concurrent I/O operations? Depending on whether you have a hard disk drive or solid state drive and what the characteristics of that drive is, concurrency might even hurt your I/O performance. You may get some performance improvement from the Pandas operations if they release the GIL.

Multiprocessing, which does provide true parallelism of CPU-intensive functions, has its own overhead in the creation of the processes and in passing data from one address space to another (i.e. one process to another). This additional overhead that you do not have in serial processing has to be compensated for by the savings achieved by parallelizing your calculations. It's not clear from what you have shown, if that is indeed representative of your actual situation, that you would gain anything from that parallelism. But, then, of course, you would not have to worry about the thread safety of deepcopy since once each process has a copy of raw_data that process would be running a single thread with its own copy of memory totally isolated from one another.

Summary

1. In general, deepcopy is not thread safe for mutable objects but since your object does not appear to be "mutating", it shouldn't be an issue. But if running under multithreading, you could do the deepcopy operation as an atomic operation under control of a multithreading.Lock without any significant loss in performance.

2. If you are using multiprocessing, and assuming raw_data was not being implemented in shared memory, then each process would be working on its own copy of raw_data to begin with. So even if another process were "mutating" raw_data, as long as any one process was running a single thread, there is no need to worry about the thread safety of deepcopy.

3. It's not clear whether multithreading or multiprocessing will achieve any performance improvements based on the code I have seen.

Benchmark

This benchmarks serial, multithreading and multiprocessing. Perhaps with only 2 keys in each dictionary this is not a realistic example but it gives a general idea:

Essentially, what happens here is that the output of the net is being sliced to get the desired part of the Q table.

The (somewhat confusing) index of [np.arange(0, self.batch_size), action] indexes each axis. So, for axis with index 1, we pick the item indicated by action. For index 0, we pick all items between 0 and self.batch_size.

If self.batch_size is the same as the length of dimension 0 of this array, then this slice can be simplified to [:, action] which is probably more familiar to most users.

Babeltrace co-maintainer here.

Indeed, Babeltrace 1 reuses the same event record object for each iteration step. This means you cannot keep an "old" event record alive as its data changes behind the scenes.

The Python bindings of Babeltrace 1 are rudimental wrappers of the library objects. This means the same constraints apply. Also, Babeltrace 1 doesn't offer any event record object copying function, so anything like copy.copy() will only copy internal pointers which will then exhibit the same issue.

Babeltrace (1 and 2) iterators cannot go backwards for performance reasons (more about this below).

The only solution I see is making your own event record copying function, keeping what's necessary in another instance of your own class. After all, you probably only need the name, timestamp, and some first-level fields of the event record.

But Babeltrace 2 is what you're looking for, especially since we don't maintain Babeltrace 1 anymore (except for critical/security bug fixes).

Babeltrace 2 offers a rich and consistent C API where many objects have a reference count and therefore can live as long as you like. The Babeltrace 2 Python bindings wrap this C API so that you can benefit from the same features.

While the C API documentation is complete, unfortunately the Python bindings one is not. However, we have this, which at least shows some examples to get you started.

About your comment:

since it seems the events are a kind of linked list where one could walk backward

No, you cannot. This is to accomodate limitations of some trace formats, in particular CTF (the format which LTTng uses). A CTF packet is a sequence of serialized binary event records: to decode event record N, you need to decode event record N - 1 first, and so on. A CTF packet can contain thousands of contiguous event records like this, CTF data streams can contain thousands of packets, and a CTF trace can contain many data streams. Knowing this, there would be no reasonable way to store the offsets of all the encoded CTF event records so that you can iterate backwards without heavy object copies.

What you can do however with Babeltrace 2 is keep the specific event record objects you need, without any copy.

In the future, we'd like a way to copy a message iterator, duplicating all its state and what's needed to continue behind the scenes. This would make it possible to keep "checkpoint iterators" so that you can go back to previous event records if you can't perform your analysis in one pass for some reason.

Note that you can also make a message iterator seek a specific timestamp, but "fast" seeking is not implemented as of this date in the ctf plugin (the iterator seeks the beginning of the message sequence and then advances until it reaches the requested timestamp, which is not efficient).

If I understand you correctly, there is only one condition you need to change. Use the % (modulo) operator to compute the remainder of an integer division:

if 0 < xx < self.width and 0 < yy < self.height and self.board[yy][xx]:

The short answer is that there's "behavior" (bug?) in the handling of added tokens for Bart (and RoBerta, GPT2, etc..) that explicitly strips spaces from the tokens adjacent (both left and right) to the added token's location. I don't see a simple work-around to this.

Added tokens are handled differently in the transformer's tokenizer code. The text is first split, using a Trie to identify any tokens in the added tokens list (see tokenization_utils.py::tokenize()). After finding any added tokens in the text, the remainder is then tokenized using the existing vocab/bpe encoding scheme (see tokenization_gpt2.py::_tokenize())

The added tokens are added to the self.unique_no_split_tokens list which prevents them from being broken down further, into smaller chunks. The code that handles this (see tokenization_utils.py::tokenize() explicitly strips the spaces from the tokens to the left and right.

You could manually remove them from the "no split" list but then they may be broken down into smaller sub-components.

Note that for "special tokens", if you add the token inside of the AddedToken class you can set the lstrip and rstrip behaviors but this isn't available for non-special tokens.

See https://github.com/huggingface/transformers/blob/v4.12.5-release/src/transformers/tokenization_utils.py#L517 for the else statement where the spaces are stripped.

There are some fundamental mistakes you have made in the code. Let me try to put some light on those first , using your lines of code

The missing logs entries from Cloud Functions is not something usually expected and could be caused by termination of the process before it reach the phase where the logs are forwarded to Cloud Monitoring. The missing log entry is not typically a concern but could indicate some unbalanced configuration. The premature resource termination can be caused by exhausting some limit. It looks like your function takes significant amount of time.

This is one of the classic Python stumbles. Your get the value of the variable, and the variable ends up with the final value.

You can do what you want by "capturing" the value of the variable as a default: