json_schema | A JSON Schema V4 and Hyperschema V4 parser and validator | JSON Processing library

For casting a map to a json part: after asking a colleague, I understood that such casting couldn't work, simply because map type is key value one without any specific schema not like struct type. Because more information is needed, map to struct cast can't work.

For the loading a json part: I managed to solve the json issue after removing the json loading and using the "failfast" mode to load the json:

You should see you other question.

Parsing Json in a string column to a struct column (and then into separate columns) can be easily done using the F.from_json function.

In your case, you need to do:

Jsonschema support was not added to the Confluent Schema Registry until version 6.0, and this is why the error reports issues about a schemaType field, because any lower versions of the Registry response/request payload do not know about that field.

Upgrading to at least that version, or using the latest version of the image will solve that error

If you just want to produce JSON, then you don't need the Registry. More details at https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/ .
You can use the regular producer.py example and provide JSON objects as strings on the CLI

Consider below approach

[EDIT 1: Added acks_late]

There are two things that can go wrong when you send a task to a Celery worker:

1. Connection issues with the broker and Message Queue.
2. Exceptions raised on the worker.

The first issue can be solved by defining retry and retry_policy as you did.

The second kind (which is what you want to solve), can be solved by calling self.retry() upon a task failure.

Depending on your type of problem, it might be helpful to set CELERY_ACKS_LATE = True.

Check out these links for further information:

Retry Lost or Failed Tasks (Celery, Django and RabbitMQ)

https://coderbook.com/@marcus/how-to-automatically-retry-failed-tasks-with-celery/

the minimum keyword does not apply to strings. it might be possible to do this with regex patterns, but it would be an ugly, bad implementation. this requirement is not well-suited for JSON schema and is better handled by an application's business logic.

First you should enter your value as int instead of string.
Then you're checking the schema of your whole json object, from which value is a key/property:

by defining schema that matches your JSON, I can read it easily