gevent | Coroutine-based concurrency library for Python | Reactive Programming library
kandi X-RAY | gevent Summary
kandi X-RAY | gevent Summary
Coroutine-based concurrency library for Python
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Patches the current thread .
- Run the setup .
- Execute a callback .
- Send a response .
- Updates all registered modules .
- Run prereleaser .
- Handle the UUID and group identifiers .
- Returns a named attribute .
- Run the callbacks .
- Given a host and port information return the result .
gevent Key Features
gevent Examples and Code Snippets
Lines of Code : 50License : Permissive (MIT)
Copy# psycopg2/app.py
from gevent import monkey
monkey.patch_all()
import os
import psycopg2
import requests
from flask import Flask, request
api_port = os.environ['PORT_API']
api_url = f'http://slow_api:{api_port}/'
app = Flask(__name__)
@app.rout # flask_app/pywsgi.py
from gevent import monkey
monkey.patch_all()
import os
from gevent.pywsgi import WSGIServer
from app import app
http_server = WSGIServer(('0.0.0.0', int(os.environ['PORT_APP'])), app)
http_server.serve_forever()
# Build and s # flask_app/patched.py
from gevent import monkey
monkey.patch_all()
from app import app # re-export
# Build and start app served by uWSGI + gevent
$ docker-compose -f async-gevent-uwsgi.yml build
$ docker-compose -f async-gevent-uwsgi.yml up
$ ab
import collections
import errno
import re
import hashlib
import base64
from base64 import b64encode, b64decode
import socket
import struct
import logging
from socket import error as SocketError
import gevent
from gunicorn.workers.base_async import import requests
from celery import task
@task(ignore_result=True)
def urlopen(url):
print(f'Opening: {url}')
try:
requests.get(url)
except requests.exceptions.RequestException as exc:
print(f'Exception for {url}: {exc!r #!/usr/bin/env python
"""
For testing: test to make sure that everything still works when gevent monkey
patches are applied.
"""
from gevent.monkey import patch_all
from prompt_toolkit.eventloop.defaults import create_event_loop
from prompt_toolkit. gunicorn -w 4 --threads 100 -b 0.0.0.0:5000 your_project:app
pip install gevent
gunicorn -w 4 -k gevent --worker-connections 1000 -b 0.0.0.0:5000 your_project:app
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
python is /opt/anaconda3/bin/python
python is /usr/local/bin/python
python is /usr/bin/python
pyinstaller --hidden-import _ssl --hidden-import engineio.async_gevent --hidden-import engineio.async_eventlet --hidden-import ssl ./server_websocket.py -y
app = Flask(__name__)
app.config['SECRET_KEY'] = 'secret!Community Discussions
Trending Discussions on gevent
QUESTION
I am using Flask-SocketIO for a project and am basically trying to make it so that users can rejoin a room and "pick up where they left off." To be more specific:
- The server emits a request to the client, with a callback to process the response and a timeout of 1 second. This is done in a loop so that the request is resent if a user rejoins the room.
- A user "rejoining" a room is defined as a user joining a room with the same name as a user who has previously been disconnected from that room. The user is given their new SID in this case and the request to the client is sent to the new SID.
What I am seeing is this:
If the user joins the room and does everything normally, the callback is processed correctly on the server.
It a user rejoins the room while the server is sending requests and then submits a response, everything on the JavaScript side works fine, the server receives an ack but does not actually run the callback that it is supposed to:
...
ANSWER
Answered 2022-Apr-02 at 15:19The reason why those callbacks do not work is that you are making the emits from a context that is based on the old and disconnected socket.
The callback is associated with the socket identified by request.sid. Associating the callback with a socket allows Flask-SocketIO to install the correct app and request contexts when the callback is invoked.
The way that you coded your color prompt is not great, because you have a long running event handler that continues to run after the client goes aways and reconnects on a different socket. A better design would be for the client to send the selected color in its own event instead of as a callback response to the server.
QUESTION
environment: Django3.1, celery5.2 Django setting.py
...ANSWER
Answered 2022-Mar-17 at 06:32# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
QUESTION
When i set my airflow on kubernetes infra i got some problem. I refered this blog. and some setting was changed for my situation. and I think everything work out but I run dag manually or scheduled. worker pod work nicely ( I think ) but web-ui always didn't change the status just running and queued... I want to know what is wrong...
here is my setting value.
Version info
...ANSWER
Answered 2022-Mar-15 at 04:01the issue is with the airflow Docker image you are using.
The ENTRYPOINT I see is a custom .sh file you have written and that decides whether to run a webserver or scheduler.
Airflow scheduler submits a pod for the tasks with args as follows
QUESTION
I just switched from ForkPool to gevent with concurrency (5) as the pool method for Celery workers running in Kubernetes pods. After the switch I've been getting a non recoverable erro in the worker:
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
The broker logs gives basically the same message:
2021-11-01 22:26:17.251 [warning] <0.18574.1> Consumer None4 on channel 1 has timed out waiting for delivery acknowledgement. Timeout used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more
I have the CELERY_ACK_LATE set up, but was not familiar with the necessity to set a timeout for the acknowledgement period. And that never happened before using processes. Tasks can be fairly long (60-120 seconds sometimes), but I can't find a specific setting to allow that.
I've read in another post in other forum a user who set the timeout on the broker configuration to a huge number (like 24 hours), and was also having the same problem, so that makes me think there may be something else related to the issue.
Any ideas or suggestions on how to make worker more resilient?
...ANSWER
Answered 2022-Mar-05 at 01:40For future reference, it seems that the new RabbitMQ versions (+3.8) introduced a tight default for consumer_timeout (15min I think).
The solution I found (that has also been added to Celery docs not long ago here) was to just add a large number for the consumer_timeout in RabbitMQ.
In this question, someone mentions setting consumer_timeout to false, in a way that using a large number is not needed, but apparently there's some specifics regarding the format of the configuration for that to work.
I'm running RabbitMQ in k8s and just done something like:
QUESTION
I'm trying to install uwsgi for a django project inside a virtual environment; I'm using windows 10.
I did pip install uwsgi & I gotCommand "python setup.py egg_info".
So to resolve the error I followed this SO answer
As per the answer I installed cygwin and gcc compiler for windows following this.
Also changed the os.uname() to platform.uname()
And now when I run `python setup.py install``. I get this error
...ANSWER
Answered 2022-Feb-16 at 14:29Step 1: Download this stable release of uWSGI
Step 2: Extract the tar file inside the site-packages folder of the virtual environment.
For example the extracted path to uwsgi should be:
QUESTION
- Possibility to run asyncio coroutines.
- Correct celery behavior on exceptions and task retries.
- Possibility to use aioredis lock.
So, how to run async tasks properly to achieve the goal?
What is RuntimeError: await wasn't used with future (below), how can I fix it?
I have already tried:
1. asgirefasync_to_sync (from asgiref https://pypi.org/project/asgiref/).
This option makes it possible to run asyncio coroutines, but retries functionality doesn't work.
2. celery-pool-asyncio(https://pypi.org/project/celery-pool-asyncio/)
Same problem as in asgiref. (This option makes it possible to run asyncio coroutines, but retries functionality doesn't work.)
3. write own async to sync decoratorI have performed try to create my own decorator like async_to_sync that runs coroutines threadsafe (asyncio.run_coroutine_threadsafe), but I have behavior as I described above.
Also I have try asyncio.run() or asyncio.get_event_loop().run_until_complete() (and self.retry(...)) inside celery task. This works well, tasks runs, retries works, but there is incorrect coroutine execution - inside async function I cannot use aioredis.
Implementation notes:
- start celery command:
celery -A celery_test.celery_app worker -l info -n worker1 -P gevent --concurrency=10 --without-gossip --without-mingle - celery app:
ANSWER
Answered 2022-Feb-04 at 07:59Maybe it helps. https://github.com/aio-libs/aioredis-py/issues/1273
The main point is:
replace all the calls to get_event_loop to get_running_loop which would remove that Runtime exception when a future is attached to a different loop.
QUESTION
I am working with a Python server which spawns a greenlet for each connection to the server. Currently, the server doesn't make use of a greenlet pool. While it was my hunch that using a pool would improve performance (mainly response time and requests-per-second throughput), in my trial-and-error implementing a pool of greenlets, there doesn't seem be much performance benefit over just using Gevent.spawn() for each greenlet/connection.
I have seen this question, which is helpful, although I am curious about the application of a greenlet pool, like Gevent Pool, in a server. Is this a useful pattern, a la thread pool? Or, does using a Pool not matter in the case of a server, since Greenlets are so lightweight compared with threads?
...ANSWER
Answered 2022-Jan-14 at 14:47Greenlets are lightweight but they do consume memory. So, even though the number of greenlets a process can support is going to be much larger than the number of threads the OS can support, there is still a cost to them. So a pool is still a useful tool for limiting the number of greenlets that can be spawned - but its size would likely be best set considerably larger than a limit for actual threads would be.
Also, due to their cooperative multitasking nature, the latency on each request (assuming each new request is handled by a new greenlet) would start to rise as the number of greenlets increases beyond a certain threshold. There's a tradeoff between allowing more requests at once and creating poor UX when each request takes an increasing amount of time to complete. It's sometimes better to cap your incoming load and reject new requests - and a pool is a useful way to do that.
QUESTION
For input/output(IO) bound we need to use async code and django is not async by default, but we can achieve this running gunicorn with the gevent worker and monkey patching:
...ANSWER
Answered 2021-Dec-31 at 07:32Do i still need to monkey patch my app or it's done by default from a worker ?
No need to patch anything in your code. No need to modify codes at all.How did gevent achieve async functionality for my django code ?
gunicornpatches everything.If we use this configuration for i/o bound, does it work? When one thread is waiting because of i/o, will the other thread be able to work?
This configuration works for i/o bound. Threads can switch between themselves at anytime (switching controlled by ultimately the operating system), no matter whether the current thread is doing I/O or CPU-bound computation. Multiple threads can work simultaneously on multi-thread CPUs. In contrast, greenlets are more of coroutines rather than threads. If a coroutine gets blocked by I/O, it actively allows another coroutine to take control of the CPU and do non-I/O stuff.I see the point in (3) (if I'm right) because of the wait time in i/o, but if this is a CPU bound, how in our case will the second thread help us or will it only help if the core is not fully loaded by one thread and there is room for another to run?
For a purely CPU-bounded task on a single-thread CPU, extra threads makes little sense.Are (3) and (4) useless because of GIL?
GIL forbids your Python codes running concurrently, but gunicorn mostly uses its libraries not written in Python. You cannot run your Django codes (in Python) with multiple threads, but the I/O tasks (handled by gunicorn, not in Python) may go concurrently. If you do need CPU utilization, use multiple processes (workers=2 * CPU_THREADS + 1) instead of multiple gthreads, or consider non-CPython interpreters likepypy, which is not constrained by GIL, but may not be compatible with your codes.
QUESTION
i using VSCode as my IDE for development odoo and for now run using Start > Debugging ( F5)
While running at web browser localhost:8069 ( default ) then appear Internal Server Error and in terminal VSCode there are errors :
...ANSWER
Answered 2021-Dec-27 at 17:01After trying for a few days and just found out that pip and python in the project are not pointing to .venv but to anaconda due to an update. when error
no module stdnum
actually there is a problem with pip so make sure your pip path with which pip or which python
- to solve .venv that doesn't work by deleting the .venv folder, create venv in python, and install all requirements again
QUESTION
Good day
I am getting an error while importing my environment:
...ANSWER
Answered 2021-Dec-03 at 09:22Build tags in you environment.yml are quite strict requirements to satisfy and most often not needed. In your case, changing the yml file to
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install gevent
You can use gevent like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
