syncd | open source code deployment tool | Continuous Deployment library
kandi X-RAY | syncd Summary
kandi X-RAY | syncd Summary
syncd is an open source code deployment tool. It is simple, efficient, and easy to use, which can improve the work efficiency of the team.
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of syncd
syncd Key Features
syncd Examples and Code Snippets
Community Discussions
Trending Discussions on syncd
QUESTION
I have a dbt project running on Cloud Composer and all my models and snapshots are running sucessfully.
I'm having trouble generating the documentation once all the processing is finished.
The integration between dbt and cloud composer is done via airflow-dbt and I have setup a task for the DbtDocsGenerateOperator.
The DAG actually runs fine, and I can see in the log that the catalog.json
file is being written to the target folder in the correspondent cloud bucket, but the file is not there.
Doing some investigation on the GCP logging, I've notice that there's a process called gcs-syncd
that is apparently removing the file.
Wondering if anyone has had success in this integration before and was able to generate the dbt docs from cloud composer?
...ANSWER
Answered 2022-Jan-03 at 00:50The problem here is that you're writing your catalog file to a location on a worker node that is mounted to the dags folder in gcs, which airflow and cloud composer manages. Per the documentation,
When you modify DAGs or plugins in the Cloud Storage bucket, Cloud Composer synchronizes the data across all the nodes in the cluster.
Cloud Composer synchronizes the dags/ and plugins/ folders uni-directionally by copying locally. Unidirectional synching means that local changes in these folders are overwritten.
The data/ and logs/ folders synchronize bi-directionally by using Cloud >Storage FUSE.
If you change the location of this file to /home/airflow/gcs/data/target/catalog.json, you should be fine as that syncs bi-directionally
QUESTION
We created a Composer v2 environment to migrate from Google Cloud Composer v1.
All DAG code was adjusted and we are using the to this date newest available image composer-2.0.0-preview.5-airflow-2.1.4
.
We noticed that even though the CPU is relaxed and memory is plenty, the Web server health
is flaky (red / green alternating every couple of minute on the environment monitoring page).
For a test I removed the K8s health check on the webserver pod in K8s (and the startup probe as well). I then found that there is a call coming from the IP of the airflow-monitoring
pod (10.63.129.6
), and shortly thereafter the gunicorn process receives a HUP:
ANSWER
Answered 2021-Nov-18 at 17:42If you have configured core:default_timezone
airflow configuration, environment health status is just a metric and it will not have any impact on the actual job/tasks execution.
You can ignore the health status or you can remove the configuration to accept default UTC timezone.
This is because Composer runs a liveness DAG named airflow_monitoring every 5 minutes and reports environment health as follow:
- When the DAG run finishes successfully the health status is True.
- If the DAG run fails, the health status is False.
- If the DAG does not finish, Composer polls the DAG’s status every 5 minutes and reports False if the one-hour timeout occurs.
QUESTION
Lets say we have 2 vectors V and W of n length. I launch a kernel in SYCL that performs 3 iterations of a for loop on each entity of V. the description of for loop is as follows:
First the loop computes the value of W at an index (W[idx]) based on 4 random values of V at the current iteration. i.e., W[idx] = sum (V[a] + V[b] + V[c]+ V[d]). where a,b,c, and d are not contiguous but are defined for each idx.
Updates V[idx] based on W[idx]. However, this update of V[idx] should be done only after the value at V[idx] has been used in step 1 to compute W.
Lets say I have 3 iterations of the for loop in the kernel. If one thread is in iteration 1 and tries to use V[2] of iteration 1 to compute W[idx = 18] at iteration 1. Another thread lets say is in iteration 2 and tries to compute W[2] in iteration 2 from a,b,c,d and computes V[2] already at iteration2.
If the second thread is ahead of first thread, the second thread will update the value of V[2] at the iteration 2. In that case, when first thread wants to use the V[2] of first iteration, how do I make sure that this is Syncd. in SYCL. Will using atomic_ref help in this case, considering that the second thread is aiming to write V[2] only after it has been used by thread [1]. Also to be noted that this V[2] of first iteration is also required to compute some other W's as well in the first iteration running in some other threads as well. How do I ensure that the value of V[2] in the second iteration gets updated in the second iteration, only when V[2] of first iteration has been used in all the required instances? Here is the source code:
...ANSWER
Answered 2021-Aug-30 at 00:36First of all, it is important to understand the synchronization guarantees that SYCL makes. Like many other heterogeneous models (e.g. OpenCL), SYCL only allows synchronization within a work group, not with work items from other work groups. The background here is that the hardware, driver, or SYCL implementation is not required to execute the work groups in parallel such that they make independent forward progress. Instead, the stack is free to execute the work groups in any order - in an extreme case, it could execute the work groups sequentially, one by one. A simple example is if you are e.g. on a single-core CPU. In that case, the backend thread pool of the SYCL implementation is probably just of size 1, and so the SYCL implementation might just iterate across all your work groups sequentially.
This means that it is very difficult to formulate producer-consumer algorithms [where one work item produces a value that another work items waits for] that span multiple work groups because it could always happen that the producer work group is scheduled to run after the consumer work group, potentially leading to a dead lock if available hardware resources prevent both from running simultaneously.
The canonical way to achieve synchronization across all work items of a kernel is therefore to split the kernel in two kernels, as you have done.
I'm not sure if you did this just for the code example or if it's also in your production code, but I'd like to point out that the q.wait()
calls between and after the kernels seem unnecessary. queue::wait()
causes the host thread to wait for completion of submitted operations, but for this use case it's sufficient if you know that the kernels run in-order.
The SYCL buffer-accessor model would automatically guarantee this because the SYCL implementation would detect that both kernels read-write vec_star
, therefore a dependency edge is inserted in the SYCL task graph.
In general, for performance you want to avoid host synchronization unless absolutely necessary, and let the device work through all enqueued work asynchronously.
Tricks you can try
In principle, in some special cases you can maybe try other approaches. However, I don't expect them to be a better choice for most use cases than just using two kernels.
group_barrier
: If you somehow manage to formulate the problem such that producer-consumer dependencies don't cross boundaries between two work groups, you can usegroup_barrier()
for synchronizationatomic_ref
: If you somehow know that your SYCL implementation/driver/hardware all guarantee that your producer work groups execute before or during the consumer work groups, you could have an atomic flag in global memory to store whether the value was already updated. You can useatomic_ref
store/load to implement something like a spin lock in global memory.- Multiple buffers: It might be possible to combine the two kernels if at the end of the second kernel you store your updated
vec
in a temporary buffer instead of the original one. After the two kernels have completed, you flip the original and temporary buffers for the next iteration.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install syncd
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page