foundry | model factory library | Web Framework library

As it turns out, this behavior is not caused by @incremental. It can be observed in a regular transformation too:

Some of the more powerful parts of Typescript's type-system allow us to specify a type-safe version of this aggregation function. I detail one approach to this in Part 1 below.

When exposing this type-safe aggregation function, we need to perform some additional checks because Functions doesn't currently support union types (see 'Functions API: Input and output types' in your Palantir documentation). I list my code for this in Part 2 below.

First, we list all the property API names that we'd like to group-by as a new union type of literal types. These should be the API names you'd normally use to access these properties in a Function.

The @configure and @incremental are set during the CI execution, while the actual code inside the function annotated by @transform_df or `@transform happens at build time.

This means that you can't programatically switch between them after the CI has passed. What you can do however is have a constant or configuration within your repo, and switch at code level whenever you want to switch these. Please make sure you understand how semantic versioning works before attempting this I.e.:

The sub-folder of this x-form file was missing in the pipeline.py. I added the missing x-form sub-folder and it worked. And just so other devs don't have to update this file every time they add a new sub-folder, I moved all the sub folders to /transforms and changed pipeline.py to what I have below & it works too!

Foundry supports the OAuth2 Authorization Code grant which provides delegated access to Foundry resources and services.

Support for the OAuth2 Client Credentials grant, which allows service user access, is under limited release with a few customers. If it’s not available for you please reach out through your internal support channels. A general release is expected early this spring, I'll try and update here if that timeframe slips.

You're on the right track with the service context path - in this case it's multipass - which is Foundry's authentication service.

Try this:

Whether the executor / driver is OOMing can often be determined through inspecting the build.

When it is likely that the executor is OOMing, the build should show the following error:

When it is likely that the driver is OOMing, the build should show the following error:

Outside of this, a couple of rules that can generally be applied are as follows:

• If the shuffle output location is missing, it is likely that the executor is OOMing.
• If the executor is lost, it is likely that the executor is OOMing.
• If the module is unreachable, it is likely that the driver is OOMing

Skew problems originate from anything that causes an exchange in your job. Things that cause exchanges include but are not limited to: joins, windows, groupBys.

These operations result in data movement across your Executors based upon the found values inside the DataFrames used. This means that when a used DataFrame has many repeated values on the column dictating the exchange, those rows all end up in the same task, thus increasing its size.

Let's consider the following example distribution of data for your join:

With help from nicojs, I was able to get basic Stryker functionality working on my library project.

As it turns out I was missing some critical configuration.

I was able to set the testRunner as karma. I added karma.ngConfig.testArguments.project and disableTypeChecks values in stryker.conf.json