GLUE | linked unified embedding for single-cell multi | Genomics library

When switching from Glue 2.0 to 3.0, which means also switching from Spark 2.4 to 3.1.1, my jobs start to fail when processing timestamps prior to 1900 with this error:

I've created a dynamic column name w/ dplyr::mutate() based on this thread Use dynamic variable names in `dplyr` and now I want to sort the new column.... but I'm not correctly passing the column name

I would like to change the values in a specific column to include information from another column using the glue function.

I do it normally like this:

I am having a lot of issues handling concurrent runs of a StateMachine (Step Function) that does have a GlueJob task in it.

The state machine is initiated by a Lambda that gets trigger by a FIFO SQS queue.

The lambda gets the message, checks how many of state machine instances are running and if this number is below the GlueJob concurrent runs threshold, it starts the State Machine.

The problem I am having is that this check fails most of the time. The state machine starts although there is not enough concurrency available for my GlueJob. Obviously, the message the SQS queue passes to lambda gets processed, so if the state machine fails for this reason, that message is gone forever (unless I catch the exception and send back a new message to the queue).

I believe this behavior is due to the speed messages gets processed by my lambda (although it's a FIFO queue, so 1 message at a time), and the fact that my checker cannot keep up.

I have implemented some time.sleep() here and there to see if things get better, but no substantial improvement.

I would like to ask you if you have ever had issues like this one and how you got them programmatically solved.

Thanks in advance!

This is my checker:

I am working on Glue in AWS and trying to test and debug in local dev. I follow the instruction here https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/ to develop Glue job locally. On that post, they use Glue 1.0 image for testing and it works as it should be. However when I load and try to dev by Glue 3.0 version; I follow the guidance steps but, I can't open Jupyter notebook on :8888 like the post said even every step seems correct.

here my cmd to start a Jupyter notebook on Glue 3.0 container

This question is somehow related but not depending to this Can you color an adjacent cell in gt table in r?:

I am explicitly looking for the modification of the part of my code!

I have this example dataset with colors of the background of mpg column depending on the values applied with html.

There are a number of different Q/A's regarding this topic on SO, but none that I have been able to find that fit my use-case. I am also very surprised that RStudio / the Shiny developers themselves have not come out with some documentation on how to do this. Regardless, take this example application:

I followed this doc to call JavaScript function from my C# script in Unity to make a WebGL game.

But there is a problem if the js code contains async/await, for example:

C# script:

I'm trying to nest a function that glues together two strings within a function that uses the combined string to name a column of a dataframe. However, the problem seems to be that the glue expression is not evaluated to a string early enough. Can (and should) I force the expression to be evaluated before it is being passed on as an argument to another function?