collector | Collector python tool for scanning and information gathering | TCP library

According to the language reference, chapter 3 "Data model" (see here):

An object’s type determines the operations that the object supports (e.g., “does it have a length?”) and also defines the possible values for objects of that type. The type() function returns an object’s type (which is an object itself). Like its identity, an object’s type is also unchangeable.[1]

which, to my mind states that the type must never change, and changing it would be illegal as it would break the language specification. The footnote however states that

[1] It is possible in some cases to change an object’s type, under certain controlled conditions. It generally isn’t a good idea though, since it can lead to some very strange behaviour if it is handled incorrectly.

I don't know of any method to change the type of an object from within python itself, so the "possible" may indeed refer to the CPython function.

As far as I can see a PyObject is defined internally as a

Handling a list of collectors with arbitrary accumulator types as a flat list can’t be done in a type safe way, as it would require declaring n type variables to capture these types, where n is the actual list size.

Therefore, you can only implement the processing as a composition of operations, each with a finite number of components know at compile time, like your recursive approach.

This still has potential for simplifications, like replacing downstreamCollectors.size() == 0 with downstreamCollectors.isEmpty() or downstreamCollectors.stream().skip(1).toList() with a copying free downstreamCollectors.subList(1, downstreamCollectors.size()).

But the biggest impact has replacing the recursive code with a Stream Reduction operation:

Notice that the kind of type inference required to deduce the resulting stream type when you use flatMap, is very different from that when you use mapMulti.

When you use flatMap, the type of the resulting stream is the same type as the return type of the lambda body. That's a special thing that the compiler has been designed to infer type variables from (i.e. the compiler "knows about" it).

However, in the case of mapMulti, the type of the resulting stream that you presumably want can only be inferred from the things you do to the consumer lambda parameter. Hypothetically, the compiler could be designed so that, for example, if you have said consumer.accept(1), then it would look at what you have passed to accept, and see that you want a Stream, and in the case of getItems().forEach(consumer), the only place where the type Item could have come from is the return type of getItems, so it would need to go look at that instead.

You are basically asking the compiler to infer the parameter types of a lambda, based on the types of arbitrary expressions inside it. The compiler simply has not been designed to do this.

Other than adding the prefix, there are other (longer) ways to let it infer a Stream as the return type of mapMulti:

Make the lambda explicitly typed:

One way to solve the issue is by parameterizing the ParentDTO Class with its own children.

There are two answers to your question :

• the absolutist : indeed, the context managers will not serve their role, the GC will have to clean the mess that should not have happened
• the pragmatic : true, but is it actually a problem ? Your file handle will get released a few milliseconds later, what's the bother ? Does it have a measurable impact on production, or is it just bikeshedding ?

I'm not an expert to Python alt implementations' differences (see this page for PyPy's example), but I posit that this lifetime problem will not occur in 99% of cases. If you happen to hit in prod, then yes, you should address it (either with your proposal, or a mix of generator with context manager) otherwise, why bother ? I mean it in a kind way : your point is strictly valid, but irrelevant to most cases.

In JDK 11 and 17 Serial collector is used when there is only one CPU available. Otherwise G1 is selected

If you limit the number of CPUS available to your container, JVM selects Serial instead of the defaultG1

This is probably one of the cases where you need to pick one between being DRY and using enums.

Enums don't go very far as far as code reuse is concerned, in Java at least; and the main reason for this is that primary benefits of using enums are reaped in static code - I mean static as in "not dynamic"/"runtime", rather than static :). Although you can "reduce" code duplication, you can hardly do much of that without introducing dependency (yes, that applies to adding a common API/interface, extracting the implementation of asListString to a utility class). And that's still an undesirable trade-off.

Furthermore, if you must use an enum (for such reasons as built-in support for serialization, database mapping, JSON binding, or, well, because it's data enumeration, etc.), you have no choice but to duplicate method declarations to an extent, even if you can share the implementation: static methods just can't be inherited, and interface methods (of which getMessage would be one) shall need an implementation everywhere. I mean this way of being "DRY" will have many ways of being inelegant.

If I were you, I would simply make this data completely dynamic

I'm not terribly familiar with your stack, so this is a high-level answer to hit on your "Why". There WILL be a more specific answer for you, somewhere down the pipe (e.g. someone that can confirm whether this thread is relevant).

While I'm no Spring Daisy (or Spring dev), you bind an expression to filmMono that resolves as the query select film.* from film..... You reference that expression four times, and it's resolved four times, in separate contexts. The ordering of the statements is likely a partially-successful attempt by the lib author to lazily evaluate the expression that you bound locally, such that it's able to batch the four accidentally identical queries. You most likely resolved this by collecting into an actual container, and then mapping on that container instead of the expression bound to filmMono.

In general, this situation is because the options available to library authors aren't good when the language doesn't natively support lazy evaluation. Because any operation might alter the dataset, the library author has to choose between:

• A, construct just enough scaffolding to fully record all resources needed, copy the dataset for any operations that need to mutate records in some way, and hope that they can detect any edge-cases that might leak the scaffolding when the resolved dataset was expected (getting this right is...hard).
• B, resolve each level of mapping as a query, for each context it appears in, lest any operations mutate the dataset in ways that might surprise the integrator (e.g. you).
• C, as above, except instead of duplicating the original request, just duplicate the data...at every step. Pass-by-copy gets real painful real fast on the JVM, and languages like Clojure and Scala handle this by just making the dev be very specific about whether they want to mutate in-place, or copy then mutate.

In your case, B made the most sense to the folks that wrote that lib. In fact, they apparently got close enough to A that they were able to batch all the queries that were produced by resolving the expression bound to filmMono (which are only accidentally identical), so color me a bit impressed.

Many access patterns can be rewritten to optimize for the resulting queries instead. Your milage may vary...wildly. Getting familiar with raw SQL, or else a special-purpose language like GraphQL, can give much more consistent results than relational mappers, but I'm ever more appreciative of good IDE support, and mixing domains like that often means giving up auto-complete, context highlighting, lang-server solution-proofs and linting.

Given that the scope of the question was "why did this happen?", even noting my lack of familiarity with your stack, the answer is "lazy evaluation in a language that doesn't natively support it is really hard."

Just use the version that MassTransit depends upon, which is much earlier than v13. Upgrading past that without the proper assembly redirects is likely causing your issue.