pipelines | Example pipelines and workflows for PDAL | BPM library

The Question

How do I best execute memory-intensive pipelines in Apache Beam?

Background

I've written a pipeline that takes the Naemura Bird dataset and converts the images and annotations to TF Records with TF Examples of the required format for the TF object detection API.

I tested the pipeline using DirectRunner with a small subset of images (4 or 5) and it worked fine.

The Problem

When running the pipeline with a bigger data set (day 1 of 3, ~21GB) it crashes after a while with a non-descriptive SIGKILL. I do see a memory peak before the crash and assume that the process is killed because of a too high memory load.

I ran the pipeline through strace. These are the last lines in the trace:

I'm using the Jfrog Artifactory plugin in my Jenkins pipeline to pull some in-house utilities that the pipelines use. I specify which version of the utility I want using a parameter.

After executing the server.download, I'd like to verify and report which version of the file was actually downloaded, but I can't seem to find any way at all to do that. I do get a buildInfo object returned from the server.download call, but I can find any way to pull information from that object. I just get an object reference if I try to print the buildInfo object. I'd like to abort the build and send a report out if the version of the utility downloaded is incorrect.

The question I have is, "How does one verify that a file specified by a download spec is successfully downloaded?"

I have a requirement which is as follows:

Variable Group A, has 7 set of key=value pairs Variable Group B, has 7 set of key=value pairs.

In both cases keys are the same, values are only different.

I am asking from the user, the value of be injected in variable group B, user provides me the variable group A name.

Code snippet to perform such update is as below:

I am getting {"code": "Too many requests", "message": "Request is denied due to throttling."} from ADX when I run some batch ADF pipelines. I have came across this document on workload group. I have a cluster where we did not configured work load groups. Now i assume all the queries will be managed by default workload group. I found that MaxConcurrentRequests property is 20. I have following doubts.

1. Does it mean that this is the maximum concurrent requests my cluster can handle?

2. If I create a rest API which provides data from ADX will it support only 20 requests at a given time?

3. How to find the maximum concurrent requests an ADX cluster can handle?

I am currently learning DirectX 12 and trying to get a demo application running. I am currently stuck at creating a pipeline state object using a root signature. I am using dxc to compile my vertex shader:

I am attempting to create a CI pipeline for a WCF project. I got the CI to successfully run but cannot determine where to look for the artifact. My intent is to have the CI pipeline publish this artifact in Azure and then have the CD pipeline run transformations on config files. Ultimately, we want to take that output and store it in blob storage (that will probably be another post since the WCF site is for an API).

I also realize that I really do not want to zip the artifact since I will need to transform it anyway.

Here are my questions:

1. Where is the container that the artifact 'drop' is published to?
2. How would I publish the site to the container without making it a single file.

Thanks

I am trying to deploy an existing .Net Core application using Azure Devops by creating Build and release pipelines. The build pipeline worked fine, but I get the below error when running the release pipeline (under Deploy Azure App Service).

Error: No package found with specified pattern: D:\a\r1\a***.zip
Check if the package mentioned in the task is published as an artifact in the build or a previous stage and downloaded in the current job.

What should be done to fix this?

A similar question is already asked, but the answer did not help me solve my problem: Sklearn components in pipeline is not fitted even if the whole pipeline is?

I'm trying to use multiple pipelines to preprocess my data with a One Hot Encoder for categorical and numerical data (as suggested in this blog).

Here is my code, and even though my classifier produces 78% accuracy, I can't figure out why I cannot plot the decision-tree I'm training and what can help me fix the problem. Here is the code snippet:

I want to break down a large job, running on a Microsoft-hosted agent, into smaller jobs running sequentially, on the same agent. The large job is organized like this: