Popular New Releases in Bot
Telegram
Update to 8.7.1 (2629)
discord.js
@discordjs/voice@0.9.0
YYeTsBot
v2021-07-24
botkit
v4.15.0
beets
beets 1.6.0
Popular Libraries in Bot
by DrKLO java
19712 GPL-2.0
Telegram for Android source
by discordjs javascript
18961 Apache-2.0
A powerful JavaScript library for interacting with the Discord API
by hubotio javascript
15903 MIT
A customizable life embetterment robot.
by timgrossmann python
13182 GPL-3.0
📷 Instagram Bot - Tool for automated Instagram interactions
by tgbot-collection python
10806 MIT
🎬 人人影视bot,完全对接人人影视全部无删减资源
by howdyai typescript
10694 MIT
Botkit is an open source developer tool for building chat bots, apps and custom integrations for major messaging platforms.
by beetbox python
10638 MIT
music library manager and MusicBrainz tagger
by wilsonfreitas python
7607
A curated list of insanely awesome libraries, packages and resources for Quants (Quantitative Finance)
by EZFNDEV python
6876
With EasyFNBot you can easily create you own Fortnite Lobby Bot in less than 5 minutes which will be online forever!
Trending New libraries in Bot
by jef typescript
4794 MIT
🤖 The world's easiest, most powerful stock checker
by adiwajshing javascript
2574
Lightweight full-featured WhatsApp Web + Multi-Device API
by Hari-Nagarajan python
2197 GPL-3.0
Tool to help us buy hard to find items.
by SudhanPlayz javascript
1586 NOASSERTION
An advanced discord music bot, supports Spotify, Soundcloud, YouTube with Shuffling, Volume Control and Web Dashboard with Slash Commands support!
by brndnmtthws python
1443 NOASSERTION
ThetaGang is an IBKR bot for collecting money
by TeamUltroid python
1355 AGPL-3.0
Advanced Multi-Featured Telegram UserBot, Built in Python Using Telethon lib.
by zhycorp typescript
1223 BSD-3-Clause
A dedicated open-source Discord bot for Zhycorp based on our Discord bot template with more features. Easy to use, and with no coding required.
by takayama-lily typescript
1214 MPL-2.0
QQ机器人协议库
by iterativv python
1132 GPL-3.0
Trading strategy for the Freqtrade crypto bot
Top Authors in Bot
1
24 Libraries
1718
2
22 Libraries
18539
3
19 Libraries
1389
4
14 Libraries
130
5
14 Libraries
123
6
13 Libraries
489
7
13 Libraries
111
8
12 Libraries
163
9
12 Libraries
1344
10
10 Libraries
54
1
24 Libraries
1718
2
22 Libraries
18539
3
19 Libraries
1389
4
14 Libraries
130
5
14 Libraries
123
6
13 Libraries
489
7
13 Libraries
111
8
12 Libraries
163
9
12 Libraries
1344
10
10 Libraries
54
Trending Kits in Bot
ChatGPT is transforming all fields, including application development. You can easily add ChatGPT to your applications to provide additional intelligence.
Here are some cool open source libraries that you can leverage to add ChatGPT capabilities in your applications.
Start with Andrej Karpathy’s NanoGPT to get a feel for how GPT works at a smaller scale, even in your laptop.
ChatGPT Prompts is a great repo to help you leverage ChatGPT personas such as a translator, interviewer, JavaScript console, UI developer, IT architect, Excel sheet, stand-up comedian, philosopher, plagiarism checker, doctor, financial analyst, and many more.
If you'd like to use ChatGPT on your desktop try ChatGPT by lencx.
Up your chat game with ChatGPT apps for WhatApp, Telegram and Discord.
If you would like to use ChatGPT to compare with Google results from your browser, or integrate ChatGPT into your IDE or a general-purpose API to integrate with any other application like Telegram and others to create your custom ChatGPT integrations, below are useful repositories. Try them to build and get familiar with ChatGPT use cases.
You can build an intelligent telegram bot. It can automatically or on request send text, video, images, documents and perform different activities.
Using a Python Telegram Bot Library, you can Broadcast, Teach, Collect Leads, Search, Reply, Remind, Play, Connect, etc. To help developers, these bots act as a BotFather. Also, you can implement the pair-to-pair security protocol and end-to-end encryption to ensure that each exchange of messages between the Bot and the user is secure.
We have handpicked the top and trending Python Telegram Bot libraries for your next project below.
python-telegram-bot
- Used for building a bot in telegram easily by coupling with Python libraries like Flask, Requests, and Viz.
- Provides an asynchronous interface for the Telegram Bot API.
- The library features many high-level classes to make the development straightforward.
Telethon
- An asynchronous Python 3 MTProto library that helps interact with Telegram’s API as a user through a bot API alternative.
- If we install cryptg, the library will work faster as the encryption and decryption will be done using C instead of Python.
- If we install pillow, larger images will be automatically resized while sending photos to prevent telegram from failing with ‘Invalid Image’ messages.
pyTelegramBotAPI
- Used as the Python implementation for the telegram bot as it supports both synchronous and asynchronous methods.
- Provides functions like send_message, send_xyz, send_document, etc. And listens to all incoming messages.
- Can have an arbitrary name, but it should have only one parameter: the message.
aiogram
- Used for building bots quickly and efficiently using the available template and a fully asynchronous framework for Telegram Bot API.
- Used in Bot applications and Automation.
- Can reply into webhook, i.e., making requests in response to any latest updates.
pyrogram
- Modern, elegant, and asynchronous Telegram MTProto API framework in Python for bots and users.
- Enables you to easily interact with the main Telegram API using a user interface or a bot API alternative using Python.
- Types and methods are all type-hinted, which will enable excellent editor support.
telepot
- Helps build applications for Telegram Bot API and works on Python 2.7 and Python 3.
- Use telepot.glance() function for extracting the headline information.
- Supports synchronous and asynchronous methods of programming.
mtprotoproxy
- Fully asynchronous and can process a lot of connections.
- Not just a tool but has an API that can help customize the Telegram MTProto proxy.
- Can be used for logging, limiting access, and creating proxy farms that are hard to filter.
BrainDamage
- Used in Runtime environments, Docker applications, and Servers.
- Can destroy the active slaves, remove the stub from the host and registry entries, run shell commands on the host, and download files on a host computer.
- Used for Artificial Intelligence and Machine Learning.
informer
- Used in Docker applications, Bot, and Automation.
- Allows you to masquerade as multiple REAL users on telegram and spy on 500+ Telegram channels per account.
- Is a Telegram Mass Surveillance Bot in Python.
TeleGram-Scraper
- Used to export competitor groups, channel members, and add them to your own group or channel.
- You can scrape search results and extract the contents produced from those search results.
- Supports telegram premium API.
telebot
- Provides the best-of-its-kind API for command routing, keyboards, and inline query requests and callbacks.
- Are Concise API, supports Command routing, Middleware, Effortless bot callbacks, and Transparent File API.
- Is a highload-ready solution that has APIs that are easy to memorize and use.
mautrix-telegram
- A hybrid Matrix- telegram puppeting or relaybot bridging library.
- Has 2-factor authentication enabled for logging in with a bot token.
- Includes a simple script to help migration between different database management systems.
python-aria-mirror-bot
- A telegram bot for mirroring files on the internet to our Google Drive or Telegram.
- Supports Mirroring direct download links to Google Drive, Upload and download progress, Docker support, Download or upload speed and ETAs, Index Link support, and many more.
- Stops duplicates for all tasks except for qBittorrent and youtube-dl tasks.
TorrentLeech-Gdrive
- Based on the Pyrogram library and Leecher.
- Supports Telegram file mirroring to the cloud with its unzipping, untar, and unrar.
- Help change the rclone destination configuration on the fly.
FAQ:
1. What is the best API framework for building a Python Telegram bot?
The Python-telegram-bot library is a fantastic framework for creating Python Telegram Bots. You can use it to create and manage Telegram bots with many features. This library makes it easier to use the Telegram Bot API by simplifying its complexity. It offers features like inline mode support, webhook integration, and message handling.
2. How does the Telegram Bot API library compare to other API alternatives?
Python-telegram-bot is a popular and well-maintained choice for making Telegram bots with Python. However, other libraries offer unique features and approaches. Here are some alternatives:
- python-telegram-bot
- pyTelegramBotAPI
- aiogram
- Telepot
3. What are the main differences between the Telegram Bot API and the main Telegram API?
Developers intend to use the Telegram Bot API to build custom clients. The platform provides special user experiences. The Bot API allows for automated bots to interact with users. The Telegram Bot API is complex. It requires a deeper understanding of the Telegram protocol. The Bot API is simpler and more user-friendly.
The Telegram API is good for making different Telegram apps. The Bot API is for making bots that interact automatically. The Telegram API lets you have much control over messaging and interactions. The Bot API makes bot-specific tasks easier.
The authentication process is different for the two APIs. To use the Telegram API, you need a user session. The Bot API, on the other hand, requires API tokens for authentication. The Telegram API includes messaging, customization, and file-sharing features. The Bot API is all about bot stuff, like commands, handling messages, and inline queries.
4. How user-friendly is the interface of Python libraries for developing bots?
Python libraries make bot development easier by providing user-friendly interfaces. But the level of user-friendliness can vary based on the specific library you choose. When evaluating how easy it is to use a library's interface, think about these things:
Documentation:
Developers need clear documentation with examples to use the library effectively.
Ease of Setup:
Installing and setting up a library can make developing easier and more efficient.
Community Support:
A strong community can offer assistance, answer queries, and share tips. It enhances the overall user experience.
Abstraction:
The extent to which the library abstracts complex API calls. It handles common use cases and can greatly impact usability.
Flexibility:
A library that is both powerful and easy to use is great for beginners and experts.
5. Can one use Python as one of many programming languages to create a bot on Telegram?
You can definitely use Python as a programming language to create a bot on Telegram. Telegram offers a Bot API. It offers a straightforward way to develop bots. Python has several libraries that make it easy to interact with this API. Some commonly used libraries include:
python-telegram-bot:
This is a popular library specifically designed to create Telegram bots in Python. The interface is easy to use and simplifies the Telegram Bot API. It makes it suitable for both beginners and experienced developers.
aiogram:
This is an asynchronous library. It uses Python's asyncio framework to build Telegram bots. It is particularly useful when handling many concurrent tasks in your bot.
pyTelegramBot API:
This library offers a simpler interface for basic bot functionalities. Beginners can easily create a basic bot without worrying about complicated details.
Telepot:
Telepot is an earlier library, but you can still use it to make Telegram bots with Python. However, it might not have newer libraries' latest features and updates.
These libraries help interact with the Telegram Bot API using Python. You can use it to create bots that answer people, organize chats, and send messages.
6. Can a Chat Bot or ChatGPT bot be built using Python libraries?
It is possible to build a Chat Bot or ChatGPT bot using Python libraries. To create a chatbot, you can use Python. It doesn't have to be complex or need advanced techniques. Here are a few ways that can help you with the same:
Rule-Based Chatbots:
You can build a basic rule-based chatbot using nltk and spaCy. These provide tools for tokenization and part-of-speech. It can help analyze user input and generate responses based on predefined rules.
Generative Chatbots:
Creating a complete model can complicate things. But with the Open AI GPT-3 API, you can add a strong language model to your Python chatbot. OpenAI's API allows you to send prompts to GPT-3 and receive generated responses. It enables you to generate more human-like interactions. Using GPT-3 will need an API key and an understanding of API integration.
Simple Retrieval-Based Chatbots:
You can use Python to create a basic chatbot that gives set responses to input. You can use libraries like gensim and scikit-learn to create models. These models match user queries with responses. These models help determine text similarity.
Custom Machine Learning Models:
For more advanced chatbots, you can use libraries like transformers. It helps fine-tune pre-trained language models for specific chatbot tasks. This approach requires some familiarity with machine learning and natural language processing concepts.
Bot Frameworks:
Some bot frameworks use Python and have extra features for creating conversational AI. Rasa allows you to create rule-based and machine learning-based chatbots. It provides tools to train and deploy them.
7. Are there any tutorials for programmers who want to make bots on Telegram with Python?
There are tutorials for programmers who want to make bots on Telegram with Python. The tutorials explain how to create a bot, including setup and message handling. They also teach you how to add more advanced features. Here are a few resources to get you started:
Python Telegram Bot Library Documentation:
The official documentation for the 'python-telegram-bot' library provides a comprehensive guide. It helps build bots on Telegram using Python. It covers installation, message handling, inline queries, basic usage, and keyboard markup.
Real Python Tutorial:
Real Python has a tutorial. It guides you through building a weather bot for Telegram using Python. It covers using the 'python-telegram-bot' library and integrating with a weather API.
YouTube Video Tutorials:
YouTube hosts a wealth of video tutorials on building Telegram bots with Python. Search for keywords like 'Telegram Bot Python tutorial' to find relevant videos. It suits your learning style.
FreeCodeCamp Tutorial:
FreeCodeCamp offers a step-by-step tutorial on building a Telegram bot using Python. You will learn how to set up a bot, handle updates, create an interactive bot, and send messages.
Dev.to Articles:
You can find articles and tutorials about making Telegram bots with Python on Dev.to. These articles cover various topics, from basic bot creation to more advanced features.
GitHub Repositories:
Developers frequently share their Telegram bot projects on GitHub. They include the source code and documentation. Exploring these repositories can provide practical examples and insights.
8. How difficult is learning Python to create a simple bot on Telegram?
Creating a basic bot on Telegram using Python can be fairly easy to learn. The difficulty level depends on how much you know about programming and if you use tools. It also depends on your experience with Python. Here is a general overview:
For Beginners:
- If you are new to programming, you may need time to learn Python's syntax and concepts.
- After learning Python, you can use it to explore the Telegram Bot API and Python libraries.
- We can make a bot that sends and gets messages, follows commands, and gives simple information. It can be achievable for beginners with some practice and patience.
For experienced programmers:
- Knowing programming makes learning Python and the libraries for creating Telegram bots easy.
- It can be helpful to know about APIs and making HTTP requests. You'll use the Telegram Bot API to interact with Telegram.
- If you don't know a lot, you might need more time to learn about some things. These include inline searches, connecting with other apps, and managing messages.
Beginner to intermediate programmers can create a basic bot on Telegram with Python. There are many helpful resources available to help you learn easily. These resources include documents, tutorials, and examples.
9. When creating a telegram Bot with Python, are there any special things to consider compared to Java or C++?
Creating a Telegram bot is similar in many programming languages. However, there are a few differences to think about. When making a Telegram bot, remember some important differences from other languages.
- Library Availability
- Development Speed
- Syntax and Conciseness
- Learning Curve
- Memory Management
- Asynchronous Vs. Synchronous
- Performance
- Integration with other Services
- Community and Libraries
- Deployment and Hosting
Python is a language that has simplicity, strong community support, and user-friendly availability. This makes it a popular choice for developing Telegram bots. Consider familiarity, project requirements, and team expertise when choosing a language.
10. Are there security concerns when using Python libraries to develop bots on Telegram?
You should know about security issues if you create bots on Telegram or other platforms. Python is safe, but libraries, code, and architecture can affect your bot's security. Here are some common security considerations:
- Input Validation and Sanitization
- API Token Protection
- Rate Limiting and Anti-Abuse Measures
- Bot Permissions
- Webhook Security
- Secure Coding Practices
- Library Security and Updates
- Data Privacy and Storage
- User Privacy
- Monitoring and Logging
- Secure Deployment
Build an end-to-end automated trading application to visualize financial data of stock holdings. Buy/sell stocks automatically using Python trading bot libs.
The key functions of a stock trading bot built using the Python algorithmic trading libraries are getting the data, training the model, entering and exiting trades, placing orders, and getting notified. Through backtesting on historical data and by minimizing human intervention, a Python trading bot can increase the speed of order entry and establish discipline in the volatile stock market to help traders earn profits. Since computer systems react quickly to fluctuating markets, an automated Python-based trading system can produce bids as soon as the defined trading conditions are met.
Browse below to check out the latest and trending Python libraries to build your automated trading bot:
zipline
- Backtesting and algorithm development framework for trading strategies.
- Easily integrates with live trading platforms.
- Provides a range of built-in statistical and risk analysis tools.
ta-lib
- Offers over 150 technical analysis indicators.
- Widely used in financial markets for quantitative analysis.
- Supports multiple programming languages, including Python.
freqtrade
- High-frequency trading bot with a focus on cryptocurrencies.
- Supports machine learning and AI-based strategy development.
- Offers an active community and a range of pre-built strategies.
binance-trade-bot
- Designed specifically for trading on the Binance exchange.
- Supports various trading pairs and strategies.
- Offers real-time market data and order execution.
jesse
- Python-based framework for cryptocurrency trading strategies.
- Provides a simple and clean syntax for strategy development.
- Supports multiple exchanges and backtesting.
Crypto-Signal
- Cryptocurrency trading signals and alert system.
- Offers technical analysis indicators and custom signal creation.
- Integration with popular cryptocurrency exchanges.
Binance-volatility-trading-bot
- Designed for volatility-based trading strategies on Binance.
- Focuses on profiting from price fluctuations.
- Offers various trading parameters for customization.
binance-trader
- A simple and lightweight Binance trading bot.
- Supports spot and futures trading on the Binance platform.
- Suitable for both beginners and experienced traders.
OctoBot
- Modular and extensible cryptocurrency trading bot.
- Provides a web-based dashboard for monitoring and configuration.
- Supports various technical analysis indicators.
cointrol
- Open-source cryptocurrency trading bot with a focus on Bitcoin.
- Supports multiple exchanges and customizable strategies.
- Offers risk management and trading performance analysis.
Cryptocurrency-Trading-Bots-Python-Beginner-Advance
- A collection of Python trading bot examples for all skill levels.
- Educational resource with various trading strategies.
- Suitable for beginners and experienced traders.
algobot
- Algorithmic trading bot framework with a range of features.
- Supports backtesting, live trading, and data analysis.
- Integrates with various data sources and brokers.
binance-tutorials
- Educational resources and tutorials for Binance trading.
- Covers various aspects of trading on the Binance platform.
- Suitable for traders looking to learn and improve their skills.
pybacktest
- A Python library for backtesting trading strategies.
- Supports historical price data analysis and strategy optimization.
- Provides visualizations for performance analysis.
crypto-trader
- A simple cryptocurrency trading bot for Bitcoin.
- Designed for automated trading on various exchanges.
- Offers basic trading strategies.
stock-trading-ml
- Machine learning-based stock trading bot.
- Uses historical stock data for predictive trading.
- Suitable for those interested in AI-driven trading.
tradingview-webhooks-bot
- A bot that uses TradingView webhooks for automated trading.
- Integrates with TradingView alerts to trigger trades.
- Suitable for traders using TradingView for technical analysis.
crypto-arbitrage
- Focuses on cryptocurrency arbitrage trading strategies.
- Aims to profit from price differences on different exchanges.
- Supports multiple cryptocurrency pairs.
TensorFlow-Bitcoin-Robot
- Utilizes TensorFlow for machine learning-based Bitcoin trading.
- Provides tools for developing and training trading models.
- Suitable for those with machine learning expertise.
pyjuque
- Cryptocurrency trading bot framework with a focus on flexibility.
- Supports custom strategies and risk management.
- Designed for traders with coding experience.
Titan
- Algorithmic trading bot for various asset classes, including cryptocurrencies.
- Offers machine learning capabilities and strategy customization.
- Designed for both beginners and experienced traders.
PyTrendFollow
- A trend-following trading bot framework.
- Focuses on capturing trends in various financial markets.
- Supports strategy development and live trading.
Bitcoin-Surge-Trading-Alpha
- A cryptocurrency trading bot with a focus on capturing surges in Bitcoin prices.
- Designed for short-term trading strategies.
- Offers features for risk management and portfolio tracking.
pytrader
- A simple and lightweight cryptocurrency trading bot.
- Supports multiple exchanges and basic trading strategies.
- Suitable for beginners and traders looking for a minimalistic approach.
FAQ
1. How to choose a Python trading bot library for backtesting and live trading?
When choosing a Python trading bot library, consider:-
1. versatility in strategy development,
2. access to historical and real-time data, and
3. support for various order types.
Seek libraries with intuitive APIs, risk controls, and exchange compatibility. Also, a vibrant community and regular updates are essential for long-term reliability.
2. Can Python trading bot libraries handle multiple trading pairs?
Python trading bot libraries are typically flexible and can handle multiple trading pairs. These libraries adapt to diverse markets, including cryptocurrencies, traditional stocks, and more.
3. How do Python trading bot libraries handle risk management to protect investments?
Risk management is paramount in Python trading bot libraries. Traders can apply stop-loss, position sizing, and risk controls to protect investments. Users can set preset loss limits, reducing the impact of market downturns.
4. What tools do Python trading bot libraries provide for performance analysis?
Python trading bot libraries provide tools for strategy optimization and performance analysis. Traders can backtest their strategies using historical data to fine-tune their algorithms. Performance metrics and visualization tools help assess strategy effectiveness and make informed adjustments.
5. Can Python trading bot libraries run on cloud servers or require infrastructure?
Python trading bot libraries can be deployed on cloud servers, eliminating hardware needs. This cloud-based approach offers scalability, cost-efficiency, and accessibility for users. Thus making it an attractive option for both beginners and experienced traders.
Here are some famous NodeJs Telegram Bot Libraries. Some of the use cases of NodeJs Telegram Bot Libraries include automating customer service, creating interactive experiences, automating workflow, and creating analytics dashboards.
Node.js Telegram Bot Libraries are libraries written in JavaScript that can be used to create custom Telegram bots. They provide a convenient way to write code that interacts with the Telegram Bot API, allowing developers to quickly create and deploy bots that can listen for commands, respond to messages, and perform other tasks.
Let us look at the libraries in detail below.
node-telegram-bot-api
- Has built-in support for multiple programming languages.
- Comprehensive API documentation.
- Automated message queueing system.
nlp.js
- Offers an advanced Natural Language Processing (NLP) engine.
- Highly modular and customizable.
- Built on the Node.js platform, so it is lightweight and fast.
telebot
- Provides an all-in-one solution for creating, managing, and deploying Telegram bots.
- Supports inline query and inline keyboard.
- Provides a unified API for all Telegram API endpoints.
LiveBot
- Offers a powerful webhook system.
- Has a built-in analytics system.
- Allows developers to easily deploy and manage bots on popular hosting services.
telegram-bot-api
- You do not need to know any programming language to get started.
- Makes it easier to build bots that can react quickly to incoming messages.
- Wide range of API methods for creating bots and interacting with users.
telegram-client
- You can quickly get a bot up and running with minimal effort.
- With a built-in command processor.
- Used to send out mass messages or schedule messages to be sent at specific times.
teleapi
- Provides an easy way to create and manage multiple bots using a single interface.
- Offers an intuitive GUI for quickly setting up your bots and customizing their behavior.
- Lightweight and simple library that lets you quickly and easily build bots.
BocchiBot
- Modern and extendable architecture.
- Extensive library of custom plugins.
- Integrated analytics system.
instauto
- Includes an easy-to-use query builder.
- Provides an easy and secure authentication process for users.
- Provides unique methods such as the getStarted method which can be used to quickly set up the bot and get started.
mtproto-core
- Supports the full MTProto protocol, which is a binary protocol developed by Telegram.
- Designed for both client and server sides, allowing developers to use the same library for both sides of the connection.
- Designed to be highly performant, allowing for fast and efficient communication between the bot and Telegram servers.
Python Weather API Libraries offer an interface for accessing weather data from different APIs. Easy-to-use functions help fetch current weather, historical weather data, and weather forecasts. It supports different languages and units of measurement.
Python Weather API Libraries can retrieve weather data for specific and multiple locations. It can retrieve information like humidity, precipitation, temperature, wind speed, and UV index. It offers customization options for specifying the level of granularity or detail of weather data required. It can handle exceptions or errors during the API request process. We can integrate with weather APIs like Dark Sky, Weatherbit, OpenWeatherMap, and AccuWeather.
Here are the 7 best Python Weather API Libraries for helping developers:
wttr.in:
- Is a weather service that offers weather data using a command-line interface.
- Allows you to retrieve current forecast and weather data for any location using a simple HTTP request.
- Offers various customization options like the language used for the data, the ability to specify the weather data format, and more.
- Supports various weather-related queries like sunsets, moon phases, and sunrises.
Jarvis:
- Is an open source library that offers an easy-to-use interface to create a virtual assistant like the character Jarvis in Iron Man.
- Allows you to create custom commands and responses for the virtual assistant.
- Allow easy interaction with users and perform tasks.
- Support is offered for speech-to-text and text-to-speech capabilities.
- Offers the ability to integrate with third-party services and APIs, weather services, and Wikipedia.
weewx:
- Is a Python library for interfacing with logging weather and weather station data.
- Offers a flexible framework to collect, display, and process weather data from various sources.
- Built-in support to upload data to social media and weather services platforms.
- Is an excellent choice for developers looking to create a weather station.
- Helps integrate weather data into applications.
pysteps:
- Is an open source Python library for precipitation nowcasting.
- Offers various algorithms and tools for analysis, processing, and visualization.
- Offers precipitation data with a focus on high-resolution radar data.
- Is a powerful tool for meteorologists, researchers, and others working with precipitation data.
wetterdienst:
- Is an open source library for accessing data from various German weather services.
- Offers a unified interface for accessing current, historical, and forecast weather data.
- Supports various weather data sources like Meteogroup, OpenWeatherMap, and Deutscher Wetterdienst.
- Offers an easy-to-use interface to access weather data.
pirate-weather-ha:
- Is designed to replace the DarkSky integration in Home Assistant with a modified and updated version.
- The underlying code will be significantly different from its previous versions.
- The API key can be received from the Pirate Weather Site and used for tracking usage and keeping AWS bills reasonable.
e_paper_weather_display:
- Is an open source library to display weather information on an e-paper display.
- Offers an easy-to-use interface for retrieving weather data from different sources.
- Helps display it on an e-paper display.
- Supports various weather data sources like Meteoblue and OpenWeatherMap.
- Is a valuable resource for anyone looking to build a weather display for commercial and personal use.
Here are some of the famous Go Telegram Bot Libraries. These Libraries are used for Building custom bots for customer service, Developing bots for e-commerce and Building bots for entertainment.
Go Telegram Bot Libraries are libraries written in the programming language Go which provide an interface to the Telegram Bot API. They allow developers to create and interact with Telegram bots in their own applications. They provide a range of features such as easy setup, custom commands, webhooks, and more.
Let us have a look at these libraries in detail below.
Telegram
- Provides a highly customizable environment to create bots.
- Provides a Botstore, which is a library of bots created by other developers.
- Provides excellent security for bots, with end-to-end encryption and access control.
telegram-bot-api
- Provides webhooks and inline keyboards for interaction with users.
- Supports deep linking to external websites and content within the Telegram app.
- Offers a unique set of features such as group chats, channels, and sticker sets.
TelegramBots
- Offers advanced features such as game playing, natural language processing, and automated responses.
- Supports webhooks which allow you to receive messages and updates from other services and systems.
- Allows you to customize your bots with a variety of useful features such as message templates
Telebot
- Offers secure bot creation and hosting with built-in features to protect user data.
- Designed to deliver reliable performance even when handling large volumes of requests.
- Offers advanced features such as inline keyboards, custom commands and user profiles.
telegram-bot-sdk
- Supports multiple programming languages, such as JavaScript, Python, and Java.
- Included in Telegram-bot-sdk is well-documented and easy to understand.
- Enables developers to quickly create and customize powerful bots to fit their needs.
go-tgbot
- Provides a powerful command line tool that helps developers to quickly create and manage their Telegram bots.
- Supports all the standard Telegram API features and adds several unique features such as custom keyboards, inline bot queries and more.
- Supports a wide range of databases, including MongoDB, Redis, BoltDB and more.
telegram-bot-go
- Quickly build and deploy a Telegram bot with minimal effort.
- Comes with comprehensive documentation and tutorials.
- Offers advanced features such as custom commands, inline commands, and support for multiple languages.
gotelebot
- Provides a convenient way to create custom commands for your bot.
- Supports webhooks, allowing you to create more interactive bots quickly and easily.
- Support for all the major Telegram Bot API features and extras.
Here are some famous C# Telegram Bot Libraries. Some of the use cases of C# Telegram Bot Libraries include automated Customer Support, News Aggregator, Online Ordering, Scheduling Appointments.
C# telegram bot libraries are collections of code that allow developers to create Telegram bots using the C# programming language. These libraries provide an API for developers to access the Telegram Bot API and create their own custom bots with a range of features. They also provide tools for creating interactive user experiences.
Let us look at the libraries in detail below.
Telegram.Bot
- Highly customizable and allows developers to tailor their bots for performing specific tasks.
- Provides advanced features such as message scheduling and inline keyboards.
- Built with security in mind, ensuring that your bots are secure and protected.
TelegramBotClients
- Built-in support for the Telegram Bot API.
- Provide support for different types of bots.
- Highly extensible, allowing you to add your own custom features to your bot.
NetTelegramBoApi
- Provides an easy way to integrate with third-party services.
- Supports Webhooks, allowing developers to easily receive updates from Telegram Bot API.
- Offers support for uploading and downloading files.
pmcenter
- Support for dynamic message creation and message modification.
- Support for custom commands and automated responses.
- Advanced analytics to track conversations and user interactions.
CryptoGramBot
- Uses an asynchronous event-driven programming model to ensure that the bot responds quickly and efficiently to user interactions.
- Supports a wide range of languages, including C#, VB.NET, and F#.
- Provides built-in integration with Microsoft Azure and other cloud services.
telegram-bot-groupagree
- Advanced chatbot features allow it to understand natural language input, leading to more accurate and efficient responses.
- Built-in support for payment systems, enabling easy monetization of bots.
- Uses machine learning algorithms to improve the accuracy of its responses.
Telegram.Bot.Framework
- Allows advanced message routing.
- Contains a rich set of features such as inline queries, location tracking, and message editing.
- Provides secure authentication, allowing users to connect with your bot without having to share their login details.
Telegram.BotAPI
- Provides an easy-to-use interface for developers to create bots.
- Supports asynchronous API calls, allowing developers to send multiple messages in one go.
- Allows developers to build bots that support a wide range of messaging capabilities.
Telebot
- Only C# Telegram Bot library that supports .NET Standard 2.0, making it cross-platform compatible.
- Easy to integrate bot logic into existing applications.
- Provides a powerful and intuitive way to create custom commands and actions that can be triggered from anywhere within the bot.
Twitter has long been seen as a short-text social platform. However, content creation on Twitter has scaled over the years across News, Questions, Announcements, Memes, Flash Sales, Teasers, Promos, Polls, Tips and others.
Elon Musk recently announced that Twitter would share ad revenue with content creators on the platform. This is a very interesting development and a step in the positive direction. How platforms monetize content and how the creators are compensated has been debated for a long time and are still evolving given the concentration of power with the platforms. Ad revenue sharing in a micro blogging site like Twitter can open up newer forms of journalism and other fast content opportunities.
The following are popular libraries for intelligent automation on the Twitter platform that can help with your content management, authoring, and engagement activities.
Consider the below popular open source libraries for integration with Twitter and Automation of tasks
Trending Discussions on Bot
How can you create a pop-up window in Discord that accepts an input from the user?
Python Selenium AWS Lambda Change WebGL Vendor/Renderer For Undetectable Headless Scraper
Is there a way to access the children of a CategoryChannel before it is deleted? [Discord.js]
Spring Boot WebClient stops sending requests
How do I get mobile status for discord bot by directly modifying IDENTIFY packet?
Changing Category/Channels Permissions Returns Error "Missing Permissions" - Novus/Discord.py
Error [ERR_REQUIRE_ESM]: require() of ES Module not supported
How to check if a bot can DM a user
How can I send Dynamic website content to scrapy with the html content generated by selenium browser?
Python/Selenium web scrap how to find hidden src value from a links?
QUESTION
How can you create a pop-up window in Discord that accepts an input from the user?
Asked 2022-Mar-30 at 07:14It's my first time seeing this feature from a Discord bot. I tried looking everywhere but it seems that I have failed. There's this feature from Captcha.bot Discord bot where you can accept input from a pop-up window inside Discord.
There's a button in an embedded message made by Captcha.bot where you will have to answer a Captcha test. After pressing the button, it creates a pop-up window like this.
After placing the right answer on the captcha bot, here's the aftermath of the experience.
All I want to learn is how to summon that pop-up window using Discord.js if it's even possible or at least learn how they did it.
ANSWER
Answered 2022-Mar-30 at 07:12Those are called modals, and they will be available in the next discord.js version, v14. There is already a pull request for this.
In the meantime, you can use an npm package like discord-modals or discordjs-modal.
You can find a working example with the discord-modals
package below. Don't forget to install it first using npm i discord-modals
.
1const {
2 Client,
3 Intents,
4 MessageActionRow,
5 MessageButton,
6} = require('discord.js');
7const discordModals = require('discord-modals');
8const { Modal, TextInputComponent, showModal } = discordModals;
9
10const TOKEN = 'YOUR TOKEN HERE';
11const client = new Client({
12 intents: [Intents.FLAGS.GUILDS, Intents.FLAGS.GUILD_MESSAGES],
13});
14discordModals(client);
15
16client.on('messageCreate', (message) => {
17 if (message.author.bot) return;
18
19 let button = new MessageActionRow();
20 button.addComponents(
21 new MessageButton()
22 .setCustomId('verification-button')
23 .setStyle('PRIMARY')
24 .setLabel('Open modal dialog'),
25 );
26 message.reply({
27 components: [button],
28 });
29});
30
31client.on('interactionCreate', async (interaction) => {
32 if (interaction.isButton()) {
33 if (interaction.customId === 'verification-button') {
34 const modal = new Modal() // We create a Modal
35 .setCustomId('verification-modal')
36 .setTitle('Verify yourself')
37 .addComponents([
38 new TextInputComponent()
39 .setCustomId('verification-input')
40 .setLabel('Answer')
41 .setStyle('SHORT')
42 .setMinLength(4)
43 .setMaxLength(12)
44 .setPlaceholder('ABCDEF')
45 .setRequired(true),
46 ]);
47
48 showModal(modal, {
49 client,
50 interaction,
51 });
52 }
53 }
54});
55
56client.on('modalSubmit', async (modal) => {
57 if (modal.customId === 'verification-modal') {
58 const response = modal.getTextInputValue('verification-input');
59 modal.reply(`Yay, your answer is submitted: "${response}"`);
60 }
61});
62
63client.once('ready', () => {
64 console.log('Bot v13 is connected...');
65});
66
67client.login(TOKEN);
68
QUESTION
Python Selenium AWS Lambda Change WebGL Vendor/Renderer For Undetectable Headless Scraper
Asked 2022-Mar-21 at 20:19Using AWS Lambda functions with Python and Selenium, I want to create a undetectable headless chrome scraper by passing a headless chrome test. I check the undetectability of my headless scraper by opening up the test and taking a screenshot. I ran this test on a Local IDE and on a Lambda server.
Implementation:
I will be using a python library called selenium-stealth and will follow their basic configuration:
1stealth(driver,
2 languages=["en-US", "en"],
3 vendor="Google Inc.",
4 platform="Win32",
5 webgl_vendor="Intel Inc.",
6 renderer="Intel Iris OpenGL Engine",
7 fix_hairline=True,
8 )
9
I implemented this configuration on a Local IDE as well as an AWS Lambda Server to compare the results.
Local IDE:
Found below are the test results running on a local IDE:
Lambda Server:
When I run this on a Lambda server, both the WebGL Vendor and Renderer are blank. as shown below:
I even tried to manually change the WebGL Vendor/Renderer using the following JavaScript command:
1stealth(driver,
2 languages=["en-US", "en"],
3 vendor="Google Inc.",
4 platform="Win32",
5 webgl_vendor="Intel Inc.",
6 renderer="Intel Iris OpenGL Engine",
7 fix_hairline=True,
8 )
9driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})
10
Then I thought maybe that it could be something wrong with the parameter number. I configured the command execution without the if statement, but the same thing happened: It worked on my Local IDE but had no effect on an AWS Lambda Server.
Simply Put:Is it possible to add Vendor/Renderer on AWS Lambda? In my efforts, it seems that there is no possible way. I made sure to submit this issue on the selenium-stealth GitHub Repository.
ANSWER
Answered 2021-Dec-18 at 02:01WebGL is a cross-platform, open web standard for a low-level 3D graphics API based on OpenGL ES, exposed to ECMAScript via the HTML5 Canvas element. WebGL at it's core is a Shader-based API using GLSL, with constructs that are semantically similar to those of the underlying OpenGL ES API. It follows the OpenGL ES specification, with some exceptions for the out of memory-managed languages such as JavaScript. WebGL 1.0 exposes the OpenGL ES 2.0 feature set; WebGL 2.0 exposes the OpenGL ES 3.0 API.
Now, with the availability of Selenium Stealth building of Undetectable Scraper using Selenium driven ChromeDriver initiated google-chrome Browsing Context have become much more easier.
selenium-stealth
selenium-stealth is a python package selenium-stealth to prevent detection. This programme tries to make python selenium more stealthy. However, as of now selenium-stealth only support Selenium Chrome.
Code Block:
1stealth(driver,
2 languages=["en-US", "en"],
3 vendor="Google Inc.",
4 platform="Win32",
5 webgl_vendor="Intel Inc.",
6 renderer="Intel Iris OpenGL Engine",
7 fix_hairline=True,
8 )
9driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})
10from selenium import webdriver
11from selenium.webdriver.chrome.options import Options
12from selenium.webdriver.chrome.service import Service
13from selenium_stealth import stealth
14
15options = Options()
16options.add_argument("start-maximized")
17options.add_experimental_option("excludeSwitches", ["enable-automation"])
18options.add_experimental_option('useAutomationExtension', False)
19s = Service('C:\\BrowserDrivers\\chromedriver.exe')
20driver = webdriver.Chrome(service=s, options=options)
21
22# Selenium Stealth settings
23stealth(driver,
24 languages=["en-US", "en"],
25 vendor="Google Inc.",
26 platform="Win32",
27 webgl_vendor="Intel Inc.",
28 renderer="Intel Iris OpenGL Engine",
29 fix_hairline=True,
30 )
31
32driver.get("https://bot.sannysoft.com/")
33
Browser Screenshot:
You can find a detailed relevant discussion in Can a website detect when you are using Selenium with chromedriver?
Changing WebGL Vendor/Renderer in AWS Lambda
AWS Lambda enables us to deliver compressed WebGL websites to end users. When requested webpage objects are compressed, the transfer size is reduced, leading to faster downloads, lower cloud storage fees, and lower data transfer fees. Improved load times also directly influence the viewer experience and retention, which helps in improving website conversion and discoverability. Using WebGL, websites are more immersive while still being accessible via a browser URL. Through this technique AWS Lambda to automatically compress the objects uploaded to S3.
Background on compression and WebGLHTTP compression is a capability that can be built into web servers and web clients to improve transfer speed and bandwidth utilization. This capability is negotiated between the server and the client using an HTTP header which may indicate that a resource being transferred, cached, or otherwise referenced is compressed. AWS Lambda on the server-side supports Content-Encoding header.
On the client-side, most browsers today support brotli and gzip compression through HTTP headers (Accept-Encoding: deflate, br, gzip
) and can handle server response headers. This means browsers will automatically download and decompress content from a web server at the client-side, before rendering webpages to the viewer.
Conclusion
Due to this constraint you may not be able to change the WebGL Vendor/Renderer in AWS Lambda, else it may directly affect the process of rendering webpages to the viewers and can stand out to be a bottleneck in UX.
tl; dr
You can find a couple of relevant detailed discussion in:
QUESTION
Is there a way to access the children of a CategoryChannel before it is deleted? [Discord.js]
Asked 2022-Feb-19 at 14:09I handle a channelDelete event in my discord bot. My original intent was to do the following:
- Listen for when a channel is deleted
- Check to see if its type equals 'GUILD_CATEGORY'
- Delete all the channels under that category
I can typically access channels under a CategoryChannel
through its property called children
anywhere else except during this event...
1module.exports = {
2 name: 'channelDelete',
3 once: false,
4 async execute(channel) {
5 if(channel.type == 'GUILD_CATEGORY')
6 for(let [child_id, child] of channel.children)
7 if (child.deletable) child.delete();
8 },
9}
10
I can confirm the code is being executed, but the problem is that the incoming channel object is in a state where it is already deleted, and I cannot get the children:
- During debugging, I noticed the channel object has the property
deleted: true
- The
children
property is empty, even though I know there were channels in that channel category prior to deletion
Is there a way for my bot to collect and handle the children of a CategoryChannel
prior to its deletion?
ANSWER
Answered 2022-Feb-19 at 14:09Unfortunately, this is how CategoryChannels work in discord.js...
When the category is deleted, discord.js sends a request to the API to delete the channel. Only then, Discord sends you the event after the category is deleted.
What happens next is that the children are not located in the category anymore! So you will not be able to get the children inside the CategoryChannel object.
This is the code for the children
property
1module.exports = {
2 name: 'channelDelete',
3 once: false,
4 async execute(channel) {
5 if(channel.type == 'GUILD_CATEGORY')
6 for(let [child_id, child] of channel.children)
7 if (child.deletable) child.delete();
8 },
9}
10get children() {
11 return this.guild.channels.cache.filter(c => c.parentId === this.id);
12}
13
It filters the channels which are in the guild, but those children channels are not located in the category anymore. Which is the reason the property is empty.
The (Only?) SolutionThis will require caching the channels themselves. As there aren't really any other solutions.
You may cache the channels in different ways.
Just remember... there are instances and references in Javascript, and failure to acknowledge may lead to weird behaviors. The following code only works for small bots without sharding, just so you know.
1module.exports = {
2 name: 'channelDelete',
3 once: false,
4 async execute(channel) {
5 if(channel.type == 'GUILD_CATEGORY')
6 for(let [child_id, child] of channel.children)
7 if (child.deletable) child.delete();
8 },
9}
10get children() {
11 return this.guild.channels.cache.filter(c => c.parentId === this.id);
12}
13const cachedChannels = new Map()
14
15client.on('ready', () => {
16 // Looping thru guilds...
17 client.guilds.forEach((guild) => {
18 // Filtering the channels to category channels only and looping thru them...
19 guild.channels.filter((channel) => channel.type === "GUILD_CATEGORY").forEach((category) => {
20 // Note on references and instances: Numbers, Strings and Booleans do not have instances like Object, Arrays, etc. do
21 // We are using an array here to abuse the references
22 cachedChannels.set(category.id, [])
23
24 // Looping thru the children channels...
25 category.children.forEach((children) => {
26 // This is the array we've created earlier
27 const storedChannels = cachedChannels.get(category.id)
28
29 // Pushing the ID so we can fetch them later
30 storedChannels.push(children.id)
31 })
32 })
33 })
34})
35
36client.on('channelDelete', (channel) => {
37 if (channel.type === "GUILD_CATEGORY") {
38 // Looping thru our cached channel IDs defined earlier
39 cachedChannels.get(channel.id).forEach((id) => {
40 const child = channel.guild.channels.get(id)
41 child.delete()
42 })
43
44 // Delete the cached IDs to save memory
45 cachedChannels.delete(channel.id)
46 }
47})
48
49
QUESTION
Spring Boot WebClient stops sending requests
Asked 2022-Feb-18 at 14:42I am running a Spring Boot app that uses WebClient for both non-blocking and blocking HTTP requests. After the app has run for some time, all outgoing HTTP requests seem to get stuck.
WebClient is used to send requests to multiple hosts, but as an example, here is how it is initialized and used to send requests to Telegram:
WebClientConfig:
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8
The same ReactorClientHttpConnector is used for all WebClients.
TelegramClient:
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8 @Autowired
9 ReactorClientHttpConnector httpClient;
10
11 WebClient webClient;
12
13 RateLimiter rateLimiter;
14
15 @PostConstruct
16 public void init() {
17 webClient = WebClient.builder()
18 .clientConnector(httpClient)
19 .baseUrl(telegramUrl)
20 .build();
21
22 rateLimiter = RateLimiter.of("telegram-rate-limiter",
23 RateLimiterConfig.custom()
24 .limitRefreshPeriod(Duration.ofMinutes(1))
25 .limitForPeriod(20)
26 .build());
27 }
28
29 public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
30 webClient.post().uri(String.format("/bot%s/sendMessage", token))
31 .contentType(MediaType.APPLICATION_JSON)
32 .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
33 .with("text", message))
34 .retrieve()
35 .bodyToMono(Void.class)
36 .transformDeferred(RateLimiterOperator.of(rateLimiter))
37 .block();
38 }
39
The RateLimiter is used to ensure the number of requests does not exceed 20 a minute as specified in the Telegram API.
When the app is started, all requests are resolved normally as expected. But after some time has passed, all requests seem to get stuck. The amount of time needed for this to happen can vary from a few hours to a few days. It happens for all requests to different hosts and is easily noticeable when messages from the TelegramBot stops. Once the requests get stuck, they are stuck indefinitely and I have to restart the app to get it working again.
There are no exceptions in the log that seem to have caused this. Since I maintain a queue for my telegram messages, I can see the point in time when the requests stop when the number of messages in the queue steadily increases and when errors happen in the other processes that are waiting for the requests to resolve.
It does not seem like the requests are even sent out as the connect timeout and response timeout that I have set do not take effect.
I had previously also tried setting idle time to 0 but that did not solve the problem
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8 @Autowired
9 ReactorClientHttpConnector httpClient;
10
11 WebClient webClient;
12
13 RateLimiter rateLimiter;
14
15 @PostConstruct
16 public void init() {
17 webClient = WebClient.builder()
18 .clientConnector(httpClient)
19 .baseUrl(telegramUrl)
20 .build();
21
22 rateLimiter = RateLimiter.of("telegram-rate-limiter",
23 RateLimiterConfig.custom()
24 .limitRefreshPeriod(Duration.ofMinutes(1))
25 .limitForPeriod(20)
26 .build());
27 }
28
29 public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
30 webClient.post().uri(String.format("/bot%s/sendMessage", token))
31 .contentType(MediaType.APPLICATION_JSON)
32 .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
33 .with("text", message))
34 .retrieve()
35 .bodyToMono(Void.class)
36 .transformDeferred(RateLimiterOperator.of(rateLimiter))
37 .block();
38 }
39 @Bean
40 public ReactorClientHttpConnector httpClient() {
41 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
42 HttpClient httpClient = HttpClient.newConnection()
43 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
44 .responseTimeout(Duration.ofMillis(responseTimeout));
45 return new ReactorClientHttpConnector(httpClient);
46 }
47
Update:
I enabled metrics and viewed it using micrometer when it got stuck. Interestingly, it shows that there is one connection for Telegram, but also shows no connections on idle, pending or active.
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8 @Autowired
9 ReactorClientHttpConnector httpClient;
10
11 WebClient webClient;
12
13 RateLimiter rateLimiter;
14
15 @PostConstruct
16 public void init() {
17 webClient = WebClient.builder()
18 .clientConnector(httpClient)
19 .baseUrl(telegramUrl)
20 .build();
21
22 rateLimiter = RateLimiter.of("telegram-rate-limiter",
23 RateLimiterConfig.custom()
24 .limitRefreshPeriod(Duration.ofMinutes(1))
25 .limitForPeriod(20)
26 .build());
27 }
28
29 public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
30 webClient.post().uri(String.format("/bot%s/sendMessage", token))
31 .contentType(MediaType.APPLICATION_JSON)
32 .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
33 .with("text", message))
34 .retrieve()
35 .bodyToMono(Void.class)
36 .transformDeferred(RateLimiterOperator.of(rateLimiter))
37 .block();
38 }
39 @Bean
40 public ReactorClientHttpConnector httpClient() {
41 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
42 HttpClient httpClient = HttpClient.newConnection()
43 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
44 .responseTimeout(Duration.ofMillis(responseTimeout));
45 return new ReactorClientHttpConnector(httpClient);
46 }
47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
51
Could the issue be this missing connection?
Update 2:
I thought this might be related to this other issue: Closing Reactor Netty connection on error status codes
So I updated my HttpClient to this:
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8 @Autowired
9 ReactorClientHttpConnector httpClient;
10
11 WebClient webClient;
12
13 RateLimiter rateLimiter;
14
15 @PostConstruct
16 public void init() {
17 webClient = WebClient.builder()
18 .clientConnector(httpClient)
19 .baseUrl(telegramUrl)
20 .build();
21
22 rateLimiter = RateLimiter.of("telegram-rate-limiter",
23 RateLimiterConfig.custom()
24 .limitRefreshPeriod(Duration.ofMinutes(1))
25 .limitForPeriod(20)
26 .build());
27 }
28
29 public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
30 webClient.post().uri(String.format("/bot%s/sendMessage", token))
31 .contentType(MediaType.APPLICATION_JSON)
32 .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
33 .with("text", message))
34 .retrieve()
35 .bodyToMono(Void.class)
36 .transformDeferred(RateLimiterOperator.of(rateLimiter))
37 .block();
38 }
39 @Bean
40 public ReactorClientHttpConnector httpClient() {
41 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
42 HttpClient httpClient = HttpClient.newConnection()
43 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
44 .responseTimeout(Duration.ofMillis(responseTimeout));
45 return new ReactorClientHttpConnector(httpClient);
46 }
47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
51 @Bean
52 public ReactorClientHttpConnector httpClient() {
53 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
54 .doAfterResponseSuccess((r, c) -> c.dispose())
55 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
56 .responseTimeout(Duration.ofMillis(responseTimeout));
57 return new ReactorClientHttpConnector(httpClient);
58 }
59
But all that seemed to do is accelerate the occurrence of the problem. Just like before, the active, pending and idle connections do not add up to the total connections. The total is always greater than the other 3 metrics added together.
Update 3: I did a thread dump when the issue happened. There were a total of 74 threads, so I don't think the app is running out of threads.
The dump for the Telegram thread:
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8 @Autowired
9 ReactorClientHttpConnector httpClient;
10
11 WebClient webClient;
12
13 RateLimiter rateLimiter;
14
15 @PostConstruct
16 public void init() {
17 webClient = WebClient.builder()
18 .clientConnector(httpClient)
19 .baseUrl(telegramUrl)
20 .build();
21
22 rateLimiter = RateLimiter.of("telegram-rate-limiter",
23 RateLimiterConfig.custom()
24 .limitRefreshPeriod(Duration.ofMinutes(1))
25 .limitForPeriod(20)
26 .build());
27 }
28
29 public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
30 webClient.post().uri(String.format("/bot%s/sendMessage", token))
31 .contentType(MediaType.APPLICATION_JSON)
32 .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
33 .with("text", message))
34 .retrieve()
35 .bodyToMono(Void.class)
36 .transformDeferred(RateLimiterOperator.of(rateLimiter))
37 .block();
38 }
39 @Bean
40 public ReactorClientHttpConnector httpClient() {
41 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
42 HttpClient httpClient = HttpClient.newConnection()
43 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
44 .responseTimeout(Duration.ofMillis(responseTimeout));
45 return new ReactorClientHttpConnector(httpClient);
46 }
47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
51 @Bean
52 public ReactorClientHttpConnector httpClient() {
53 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
54 .doAfterResponseSuccess((r, c) -> c.dispose())
55 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
56 .responseTimeout(Duration.ofMillis(responseTimeout));
57 return new ReactorClientHttpConnector(httpClient);
58 }
59"TelegramBot" #20 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=47154.24s tid=0x00007f6b28e73000 nid=0x1c waiting on condition [0x00007f6aed6fb000]
60 java.lang.Thread.State: WAITING (parking)
61 at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
62 - parking to wait for <0x00000000fa865c80> (a java.util.concurrent.CountDownLatch$Sync)
63 at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
64 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
65 at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1039)
66 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1345)
67 at java.util.concurrent.CountDownLatch.await(java.base@11.0.13/CountDownLatch.java:232)
68 at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
69 at reactor.core.publisher.Mono.block(Mono.java:1707)
70 at com.moon.arbitrage.cm.feign.TelegramClient.sendMessage(TelegramClient.java:59)
71 at com.moon.arbitrage.cm.service.TelegramService.lambda$sendArbMessage$0(TelegramService.java:53)
72 at com.moon.arbitrage.cm.service.TelegramService$$Lambda$1092/0x000000084070f840.run(Unknown Source)
73 at com.moon.arbitrage.cm.service.TelegramService.task(TelegramService.java:82)
74 at com.moon.arbitrage.cm.service.TelegramService$$Lambda$920/0x0000000840665040.run(Unknown Source)
75 at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
76
77 Locked ownable synchronizers:
78 - None
79
The reactor worker threads:
1 @Bean
2 public ReactorClientHttpConnector httpClient() {
3 HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
4 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
5 .responseTimeout(Duration.ofMillis(responseTimeout));
6 return new ReactorClientHttpConnector(httpClient);
7 }
8 @Autowired
9 ReactorClientHttpConnector httpClient;
10
11 WebClient webClient;
12
13 RateLimiter rateLimiter;
14
15 @PostConstruct
16 public void init() {
17 webClient = WebClient.builder()
18 .clientConnector(httpClient)
19 .baseUrl(telegramUrl)
20 .build();
21
22 rateLimiter = RateLimiter.of("telegram-rate-limiter",
23 RateLimiterConfig.custom()
24 .limitRefreshPeriod(Duration.ofMinutes(1))
25 .limitForPeriod(20)
26 .build());
27 }
28
29 public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
30 webClient.post().uri(String.format("/bot%s/sendMessage", token))
31 .contentType(MediaType.APPLICATION_JSON)
32 .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
33 .with("text", message))
34 .retrieve()
35 .bodyToMono(Void.class)
36 .transformDeferred(RateLimiterOperator.of(rateLimiter))
37 .block();
38 }
39 @Bean
40 public ReactorClientHttpConnector httpClient() {
41 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
42 HttpClient httpClient = HttpClient.newConnection()
43 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
44 .responseTimeout(Duration.ofMillis(responseTimeout));
45 return new ReactorClientHttpConnector(httpClient);
46 }
47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
51 @Bean
52 public ReactorClientHttpConnector httpClient() {
53 HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
54 .doAfterResponseSuccess((r, c) -> c.dispose())
55 .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
56 .responseTimeout(Duration.ofMillis(responseTimeout));
57 return new ReactorClientHttpConnector(httpClient);
58 }
59"TelegramBot" #20 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=47154.24s tid=0x00007f6b28e73000 nid=0x1c waiting on condition [0x00007f6aed6fb000]
60 java.lang.Thread.State: WAITING (parking)
61 at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
62 - parking to wait for <0x00000000fa865c80> (a java.util.concurrent.CountDownLatch$Sync)
63 at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
64 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
65 at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1039)
66 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1345)
67 at java.util.concurrent.CountDownLatch.await(java.base@11.0.13/CountDownLatch.java:232)
68 at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
69 at reactor.core.publisher.Mono.block(Mono.java:1707)
70 at com.moon.arbitrage.cm.feign.TelegramClient.sendMessage(TelegramClient.java:59)
71 at com.moon.arbitrage.cm.service.TelegramService.lambda$sendArbMessage$0(TelegramService.java:53)
72 at com.moon.arbitrage.cm.service.TelegramService$$Lambda$1092/0x000000084070f840.run(Unknown Source)
73 at com.moon.arbitrage.cm.service.TelegramService.task(TelegramService.java:82)
74 at com.moon.arbitrage.cm.service.TelegramService$$Lambda$920/0x0000000840665040.run(Unknown Source)
75 at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
76
77 Locked ownable synchronizers:
78 - None
79"reactor-http-epoll-1" #15 daemon prio=5 os_prio=0 cpu=810.44ms elapsed=47157.07s tid=0x00007f6b281c4000 nid=0x17 runnable [0x00007f6b0c46c000]
80 java.lang.Thread.State: RUNNABLE
81 at io.netty.channel.epoll.Native.epollWait0(Native Method)
82 at io.netty.channel.epoll.Native.epollWait(Native.java:177)
83 at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286)
84 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
85 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
86 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
87 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
88 at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
89
90 Locked ownable synchronizers:
91 - None
92
93"reactor-http-epoll-2" #16 daemon prio=5 os_prio=0 cpu=1312.16ms elapsed=47157.07s tid=0x00007f6b281c5000 nid=0x18 waiting on condition [0x00007f6b0c369000]
94 java.lang.Thread.State: WAITING (parking)
95 at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
96 - parking to wait for <0x00000000fa865948> (a java.util.concurrent.CompletableFuture$Signaller)
97 at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
98 at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
99 at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
100 at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
101 at java.util.concurrent.CompletableFuture.get(java.base@11.0.13/CompletableFuture.java:1998)
102 at com.moon.arbitrage.cm.service.OrderService.reconcileOrder(OrderService.java:103)
103 at com.moon.arbitrage.cm.service.BotService$BotTask.lambda$task$1(BotService.java:383)
104 at com.moon.arbitrage.cm.service.BotService$BotTask$$Lambda$1161/0x00000008400af440.accept(Unknown Source)
105 at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:171)
106 at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
107 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
108 at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249)
109 at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
110 at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
111 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
112 at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
113 at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
114 at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
115 at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
116 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
117 at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
118 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
119 at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
120 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
121 at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
122 at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
123 at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)
124 at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:702)
125 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
126 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
127 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
128 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
129 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
130 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
131 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
132 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
133 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
134 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
135 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
136 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
137 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
138 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
139 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
140 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
141 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
142 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
143 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
144 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
145 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
146 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
147 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
148 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
149 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
150 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
151 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
152 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
153 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
154 at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
155 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
156 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
157 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
158 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
159 at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
160
161 Locked ownable synchronizers:
162 - None
163
164"reactor-http-epoll-3" #17 daemon prio=5 os_prio=0 cpu=171.84ms elapsed=47157.07s tid=0x00007f6b28beb000 nid=0x19 runnable [0x00007f6b0c26a000]
165 java.lang.Thread.State: RUNNABLE
166 at io.netty.channel.epoll.Native.epollWait0(Native Method)
167 at io.netty.channel.epoll.Native.epollWait(Native.java:177)
168 at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
169 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
170 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
171 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
172 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
173 at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
174
175 Locked ownable synchronizers:
176 - None
177
178"reactor-http-epoll-4" #18 daemon prio=5 os_prio=0 cpu=188.10ms elapsed=47157.07s tid=0x00007f6b28b7d800 nid=0x1a runnable [0x00007f6b0c169000]
179 java.lang.Thread.State: RUNNABLE
180 at io.netty.channel.epoll.Native.epollWait0(Native Method)
181 at io.netty.channel.epoll.Native.epollWait(Native.java:177)
182 at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
183 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
184 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
185 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
186 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
187 at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
188
189 Locked ownable synchronizers:
190 - None
191
Seems like one of them is blocked with another task (that isn't even from the Telegram service) but that shouldn't be an issue since the other three worker threads are runnable right?
ANSWER
Answered 2021-Dec-20 at 14:25I would propose to take a look in the RateLimiter direction. Maybe it does not work as expected, depending on the number of requests your application does over time. From the Javadoc for Ratelimiter: "It is important to note that the number of permits requested never affects the throttling of the request itself ... but it affects the throttling of the next request. I.e., if an expensive task arrives at an idle RateLimiter, it will be granted immediately, but it is the next request that will experience extra throttling, thus paying for the cost of the expensive task." Also helpful might be this discussion: github or github
I could imaginge there is some throttling adding up or other effect in the RateLimiter, i would try to play around with it and make sure this thing really works the way you want. Alternatively, consider using Spring @Scheduled to read from your queue. You might want to spice it up using embedded JMS for further goodies (message persistence etc).
QUESTION
How do I get mobile status for discord bot by directly modifying IDENTIFY packet?
Asked 2022-Feb-09 at 15:05Apparently, discord bots can have mobile status as opposed to the desktop (online) status that one gets by default.
After a bit of digging I found out that such a status is achieved by modifying the IDENTIFY packet
in discord.gateway.DiscordWebSocket.identify
modifying the value of $browser
to Discord Android
or Discord iOS
should theoretically get us the mobile status.
After modifying code snippets I found online which does this, I end up with this :
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32
Now all there is left to do is overwrite the discord.gateway.DiscordWebSocket.identify
during runtime in the main file, something like this :
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32import discord
33import os
34from discord.ext import commands
35import mobile_status
36
37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
38bot = commands.Bot(command_prefix="?")
39
40@bot.event
41async def on_ready():
42 print(f"Sucessfully logged in as {bot.user}")
43
44bot.run(os.getenv("DISCORD_TOKEN"))
45
And we do get the mobile status successfully
But here's the problem, I wanted to directly modify the file (which held the function) rather than monkey-patching it during runtime. So I cloned the dpy lib locally and edited the file on my machine, it ended up looking like this :
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32import discord
33import os
34from discord.ext import commands
35import mobile_status
36
37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
38bot = commands.Bot(command_prefix="?")
39
40@bot.event
41async def on_ready():
42 print(f"Sucessfully logged in as {bot.user}")
43
44bot.run(os.getenv("DISCORD_TOKEN"))
45 async def identify(self):
46 """Sends the IDENTIFY packet."""
47 payload = {
48 'op': self.IDENTIFY,
49 'd': {
50 'token': self.token,
51 'properties': {
52 '$os': sys.platform,
53 '$browser': 'Discord Android',
54 '$device': 'Discord Android',
55 '$referrer': '',
56 '$referring_domain': ''
57 },
58 'compress': True,
59 'large_threshold': 250,
60 'v': 3
61 }
62 }
63 # ...
64
(edited both $browser
and $device
to Discord Android
just to be safe)
But this does not work and just gives me the regular desktop online icon.
So the next thing I did is to inspect the identify
function after it has been monkey-patched, so I could just look at the source code and see what went wrong earlier, but due to hard luck I got this error :
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32import discord
33import os
34from discord.ext import commands
35import mobile_status
36
37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
38bot = commands.Bot(command_prefix="?")
39
40@bot.event
41async def on_ready():
42 print(f"Sucessfully logged in as {bot.user}")
43
44bot.run(os.getenv("DISCORD_TOKEN"))
45 async def identify(self):
46 """Sends the IDENTIFY packet."""
47 payload = {
48 'op': self.IDENTIFY,
49 'd': {
50 'token': self.token,
51 'properties': {
52 '$os': sys.platform,
53 '$browser': 'Discord Android',
54 '$device': 'Discord Android',
55 '$referrer': '',
56 '$referring_domain': ''
57 },
58 'compress': True,
59 'large_threshold': 250,
60 'v': 3
61 }
62 }
63 # ...
64Traceback (most recent call last):
65 File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
66 print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
67 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
68 lines, lnum = getsourcelines(object)
69 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
70 lines, lnum = findsource(object)
71 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
72 raise OSError('could not get source code')
73OSError: could not get source code
74
Code :
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32import discord
33import os
34from discord.ext import commands
35import mobile_status
36
37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
38bot = commands.Bot(command_prefix="?")
39
40@bot.event
41async def on_ready():
42 print(f"Sucessfully logged in as {bot.user}")
43
44bot.run(os.getenv("DISCORD_TOKEN"))
45 async def identify(self):
46 """Sends the IDENTIFY packet."""
47 payload = {
48 'op': self.IDENTIFY,
49 'd': {
50 'token': self.token,
51 'properties': {
52 '$os': sys.platform,
53 '$browser': 'Discord Android',
54 '$device': 'Discord Android',
55 '$referrer': '',
56 '$referring_domain': ''
57 },
58 'compress': True,
59 'large_threshold': 250,
60 'v': 3
61 }
62 }
63 # ...
64Traceback (most recent call last):
65 File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
66 print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
67 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
68 lines, lnum = getsourcelines(object)
69 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
70 lines, lnum = findsource(object)
71 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
72 raise OSError('could not get source code')
73OSError: could not get source code
74import discord
75import os
76from discord.ext import commands
77import mobile_status
78import inspect
79
80discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
81print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
82bot = commands.Bot(command_prefix="?")
83
84@bot.event
85async def on_ready():
86 print(f"Sucessfully logged in as {bot.user}")
87
88bot.run(os.getenv("DISCORD_TOKEN"))
89
Since this same behavior was exhibited for every patched function (aforementioned one and the loc["identify"]
) I could no longer use inspect.getsource(...)
and then relied upon dis.dis
which lead to much more disappointing results
The disassembled data looks exactly identical to the monkey-patched working version, so the directly modified version simply does not work despite function content being the exact same. (In regards to disassembled data)
Notes: Doing Discord iOS
directly does not work either, changing the $device
to some other value but keeping $browser
does not work, I have tried all combinations, none of them work.
TL;DR: How to get mobile status for discord bot without monkey-patching it during runtime?
ANSWER
Answered 2022-Feb-07 at 23:03The following works by subclassing the relevant class, and duplicating code with the relevant changes. We also have to subclass the Client
class, to overwrite the place where the gateway/websocket class is used. This results in a lot of duplicated code, however it does work, and requires neither dirty monkey-patching nor editing the library source code.
However, it does come with many of the same problems as editing the library source code - mainly that as the library is updated, this code will become out of date (if you're using the archived and obsolete version of the library, you have bigger problems instead).
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32import discord
33import os
34from discord.ext import commands
35import mobile_status
36
37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
38bot = commands.Bot(command_prefix="?")
39
40@bot.event
41async def on_ready():
42 print(f"Sucessfully logged in as {bot.user}")
43
44bot.run(os.getenv("DISCORD_TOKEN"))
45 async def identify(self):
46 """Sends the IDENTIFY packet."""
47 payload = {
48 'op': self.IDENTIFY,
49 'd': {
50 'token': self.token,
51 'properties': {
52 '$os': sys.platform,
53 '$browser': 'Discord Android',
54 '$device': 'Discord Android',
55 '$referrer': '',
56 '$referring_domain': ''
57 },
58 'compress': True,
59 'large_threshold': 250,
60 'v': 3
61 }
62 }
63 # ...
64Traceback (most recent call last):
65 File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
66 print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
67 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
68 lines, lnum = getsourcelines(object)
69 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
70 lines, lnum = findsource(object)
71 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
72 raise OSError('could not get source code')
73OSError: could not get source code
74import discord
75import os
76from discord.ext import commands
77import mobile_status
78import inspect
79
80discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
81print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
82bot = commands.Bot(command_prefix="?")
83
84@bot.event
85async def on_ready():
86 print(f"Sucessfully logged in as {bot.user}")
87
88bot.run(os.getenv("DISCORD_TOKEN"))
89import asyncio
90import sys
91
92import aiohttp
93
94import discord
95from discord.gateway import DiscordWebSocket, _log
96from discord.ext.commands import Bot
97
98
99class MyGateway(DiscordWebSocket):
100
101 async def identify(self):
102 payload = {
103 'op': self.IDENTIFY,
104 'd': {
105 'token': self.token,
106 'properties': {
107 '$os': sys.platform,
108 '$browser': 'Discord Android',
109 '$device': 'Discord Android',
110 '$referrer': '',
111 '$referring_domain': ''
112 },
113 'compress': True,
114 'large_threshold': 250,
115 'v': 3
116 }
117 }
118
119 if self.shard_id is not None and self.shard_count is not None:
120 payload['d']['shard'] = [self.shard_id, self.shard_count]
121
122 state = self._connection
123 if state._activity is not None or state._status is not None:
124 payload['d']['presence'] = {
125 'status': state._status,
126 'game': state._activity,
127 'since': 0,
128 'afk': False
129 }
130
131 if state._intents is not None:
132 payload['d']['intents'] = state._intents.value
133
134 await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
135 await self.send_as_json(payload)
136 _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
137
138
139class MyBot(Bot):
140
141 async def connect(self, *, reconnect: bool = True) -> None:
142 """|coro|
143
144 Creates a websocket connection and lets the websocket listen
145 to messages from Discord. This is a loop that runs the entire
146 event system and miscellaneous aspects of the library. Control
147 is not resumed until the WebSocket connection is terminated.
148
149 Parameters
150 -----------
151 reconnect: :class:`bool`
152 If we should attempt reconnecting, either due to internet
153 failure or a specific failure on Discord's part. Certain
154 disconnects that lead to bad state will not be handled (such as
155 invalid sharding payloads or bad tokens).
156
157 Raises
158 -------
159 :exc:`.GatewayNotFound`
160 If the gateway to connect to Discord is not found. Usually if this
161 is thrown then there is a Discord API outage.
162 :exc:`.ConnectionClosed`
163 The websocket connection has been terminated.
164 """
165
166 backoff = discord.client.ExponentialBackoff()
167 ws_params = {
168 'initial': True,
169 'shard_id': self.shard_id,
170 }
171 while not self.is_closed():
172 try:
173 coro = MyGateway.from_client(self, **ws_params)
174 self.ws = await asyncio.wait_for(coro, timeout=60.0)
175 ws_params['initial'] = False
176 while True:
177 await self.ws.poll_event()
178 except discord.client.ReconnectWebSocket as e:
179 _log.info('Got a request to %s the websocket.', e.op)
180 self.dispatch('disconnect')
181 ws_params.update(sequence=self.ws.sequence, resume=e.resume, session=self.ws.session_id)
182 continue
183 except (OSError,
184 discord.HTTPException,
185 discord.GatewayNotFound,
186 discord.ConnectionClosed,
187 aiohttp.ClientError,
188 asyncio.TimeoutError) as exc:
189
190 self.dispatch('disconnect')
191 if not reconnect:
192 await self.close()
193 if isinstance(exc, discord.ConnectionClosed) and exc.code == 1000:
194 # clean close, don't re-raise this
195 return
196 raise
197
198 if self.is_closed():
199 return
200
201 # If we get connection reset by peer then try to RESUME
202 if isinstance(exc, OSError) and exc.errno in (54, 10054):
203 ws_params.update(sequence=self.ws.sequence, initial=False, resume=True, session=self.ws.session_id)
204 continue
205
206 # We should only get this when an unhandled close code happens,
207 # such as a clean disconnect (1000) or a bad state (bad token, no sharding, etc)
208 # sometimes, discord sends us 1000 for unknown reasons so we should reconnect
209 # regardless and rely on is_closed instead
210 if isinstance(exc, discord.ConnectionClosed):
211 if exc.code == 4014:
212 raise discord.PrivilegedIntentsRequired(exc.shard_id) from None
213 if exc.code != 1000:
214 await self.close()
215 raise
216
217 retry = backoff.delay()
218 _log.exception("Attempting a reconnect in %.2fs", retry)
219 await asyncio.sleep(retry)
220 # Always try to RESUME the connection
221 # If the connection is not RESUME-able then the gateway will invalidate the session.
222 # This is apparently what the official Discord client does.
223 ws_params.update(sequence=self.ws.sequence, resume=True, session=self.ws.session_id)
224
225
226bot = MyBot(command_prefix="?")
227
228
229@bot.event
230async def on_ready():
231 print(f"Sucessfully logged in as {bot.user}")
232
233bot.run("YOUR_BOT_TOKEN")
234
Personally, I think that the following approach, which does include some runtime monkey-patching (but no AST manipulation) is cleaner for this purpose:
1def get_mobile():
2 """
3 The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
4 Discord uses that information to know when your phone client and only your phone client has connected to Discord,
5 from there they send the extended presence object.
6 The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
7 the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
8 the mobile indicator is not triggered by the mobile client.
9 The specific values for the $os, $browser, and $device fields are can change from time to time.
10 """
11 import ast
12 import inspect
13 import re
14 import discord
15
16 def source(o):
17 s = inspect.getsource(o).split("\n")
18 indent = len(s[0]) - len(s[0].lstrip())
19
20 return "\n".join(i[indent:] for i in s)
21
22 source_ = source(discord.gateway.DiscordWebSocket.identify)
23 patched = re.sub(
24 r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
25 r"\1Discord Android\2",
26 source_,
27 )
28
29 loc = {}
30 exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
31 return loc["identify"]
32import discord
33import os
34from discord.ext import commands
35import mobile_status
36
37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
38bot = commands.Bot(command_prefix="?")
39
40@bot.event
41async def on_ready():
42 print(f"Sucessfully logged in as {bot.user}")
43
44bot.run(os.getenv("DISCORD_TOKEN"))
45 async def identify(self):
46 """Sends the IDENTIFY packet."""
47 payload = {
48 'op': self.IDENTIFY,
49 'd': {
50 'token': self.token,
51 'properties': {
52 '$os': sys.platform,
53 '$browser': 'Discord Android',
54 '$device': 'Discord Android',
55 '$referrer': '',
56 '$referring_domain': ''
57 },
58 'compress': True,
59 'large_threshold': 250,
60 'v': 3
61 }
62 }
63 # ...
64Traceback (most recent call last):
65 File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
66 print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
67 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
68 lines, lnum = getsourcelines(object)
69 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
70 lines, lnum = findsource(object)
71 File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
72 raise OSError('could not get source code')
73OSError: could not get source code
74import discord
75import os
76from discord.ext import commands
77import mobile_status
78import inspect
79
80discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
81print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
82bot = commands.Bot(command_prefix="?")
83
84@bot.event
85async def on_ready():
86 print(f"Sucessfully logged in as {bot.user}")
87
88bot.run(os.getenv("DISCORD_TOKEN"))
89import asyncio
90import sys
91
92import aiohttp
93
94import discord
95from discord.gateway import DiscordWebSocket, _log
96from discord.ext.commands import Bot
97
98
99class MyGateway(DiscordWebSocket):
100
101 async def identify(self):
102 payload = {
103 'op': self.IDENTIFY,
104 'd': {
105 'token': self.token,
106 'properties': {
107 '$os': sys.platform,
108 '$browser': 'Discord Android',
109 '$device': 'Discord Android',
110 '$referrer': '',
111 '$referring_domain': ''
112 },
113 'compress': True,
114 'large_threshold': 250,
115 'v': 3
116 }
117 }
118
119 if self.shard_id is not None and self.shard_count is not None:
120 payload['d']['shard'] = [self.shard_id, self.shard_count]
121
122 state = self._connection
123 if state._activity is not None or state._status is not None:
124 payload['d']['presence'] = {
125 'status': state._status,
126 'game': state._activity,
127 'since': 0,
128 'afk': False
129 }
130
131 if state._intents is not None:
132 payload['d']['intents'] = state._intents.value
133
134 await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
135 await self.send_as_json(payload)
136 _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
137
138
139class MyBot(Bot):
140
141 async def connect(self, *, reconnect: bool = True) -> None:
142 """|coro|
143
144 Creates a websocket connection and lets the websocket listen
145 to messages from Discord. This is a loop that runs the entire
146 event system and miscellaneous aspects of the library. Control
147 is not resumed until the WebSocket connection is terminated.
148
149 Parameters
150 -----------
151 reconnect: :class:`bool`
152 If we should attempt reconnecting, either due to internet
153 failure or a specific failure on Discord's part. Certain
154 disconnects that lead to bad state will not be handled (such as
155 invalid sharding payloads or bad tokens).
156
157 Raises
158 -------
159 :exc:`.GatewayNotFound`
160 If the gateway to connect to Discord is not found. Usually if this
161 is thrown then there is a Discord API outage.
162 :exc:`.ConnectionClosed`
163 The websocket connection has been terminated.
164 """
165
166 backoff = discord.client.ExponentialBackoff()
167 ws_params = {
168 'initial': True,
169 'shard_id': self.shard_id,
170 }
171 while not self.is_closed():
172 try:
173 coro = MyGateway.from_client(self, **ws_params)
174 self.ws = await asyncio.wait_for(coro, timeout=60.0)
175 ws_params['initial'] = False
176 while True:
177 await self.ws.poll_event()
178 except discord.client.ReconnectWebSocket as e:
179 _log.info('Got a request to %s the websocket.', e.op)
180 self.dispatch('disconnect')
181 ws_params.update(sequence=self.ws.sequence, resume=e.resume, session=self.ws.session_id)
182 continue
183 except (OSError,
184 discord.HTTPException,
185 discord.GatewayNotFound,
186 discord.ConnectionClosed,
187 aiohttp.ClientError,
188 asyncio.TimeoutError) as exc:
189
190 self.dispatch('disconnect')
191 if not reconnect:
192 await self.close()
193 if isinstance(exc, discord.ConnectionClosed) and exc.code == 1000:
194 # clean close, don't re-raise this
195 return
196 raise
197
198 if self.is_closed():
199 return
200
201 # If we get connection reset by peer then try to RESUME
202 if isinstance(exc, OSError) and exc.errno in (54, 10054):
203 ws_params.update(sequence=self.ws.sequence, initial=False, resume=True, session=self.ws.session_id)
204 continue
205
206 # We should only get this when an unhandled close code happens,
207 # such as a clean disconnect (1000) or a bad state (bad token, no sharding, etc)
208 # sometimes, discord sends us 1000 for unknown reasons so we should reconnect
209 # regardless and rely on is_closed instead
210 if isinstance(exc, discord.ConnectionClosed):
211 if exc.code == 4014:
212 raise discord.PrivilegedIntentsRequired(exc.shard_id) from None
213 if exc.code != 1000:
214 await self.close()
215 raise
216
217 retry = backoff.delay()
218 _log.exception("Attempting a reconnect in %.2fs", retry)
219 await asyncio.sleep(retry)
220 # Always try to RESUME the connection
221 # If the connection is not RESUME-able then the gateway will invalidate the session.
222 # This is apparently what the official Discord client does.
223 ws_params.update(sequence=self.ws.sequence, resume=True, session=self.ws.session_id)
224
225
226bot = MyBot(command_prefix="?")
227
228
229@bot.event
230async def on_ready():
231 print(f"Sucessfully logged in as {bot.user}")
232
233bot.run("YOUR_BOT_TOKEN")
234import sys
235from discord.gateway import DiscordWebSocket, _log
236from discord.ext.commands import Bot
237
238
239async def identify(self):
240 payload = {
241 'op': self.IDENTIFY,
242 'd': {
243 'token': self.token,
244 'properties': {
245 '$os': sys.platform,
246 '$browser': 'Discord Android',
247 '$device': 'Discord Android',
248 '$referrer': '',
249 '$referring_domain': ''
250 },
251 'compress': True,
252 'large_threshold': 250,
253 'v': 3
254 }
255 }
256
257 if self.shard_id is not None and self.shard_count is not None:
258 payload['d']['shard'] = [self.shard_id, self.shard_count]
259
260 state = self._connection
261 if state._activity is not None or state._status is not None:
262 payload['d']['presence'] = {
263 'status': state._status,
264 'game': state._activity,
265 'since': 0,
266 'afk': False
267 }
268
269 if state._intents is not None:
270 payload['d']['intents'] = state._intents.value
271
272 await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
273 await self.send_as_json(payload)
274 _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
275
276
277DiscordWebSocket.identify = identify
278bot = Bot(command_prefix="?")
279
280
281@bot.event
282async def on_ready():
283 print(f"Sucessfully logged in as {bot.user}")
284
285bot.run("YOUR_DISCORD_TOKEN")
286
As to why editing the library source code did not work for you, I can only assume that you have edited the wrong copy of the file, as people have commented.
QUESTION
Changing Category/Channels Permissions Returns Error "Missing Permissions" - Novus/Discord.py
Asked 2022-Feb-08 at 01:39SOLVED - The categories/channels were set to private which is why they couldn't be changed even with the correct permissions. My solution was to tell the user which ones and to add the role with the permissions manually. I added a message that said they can just give the bot administrator permissions and it will do it for them. Hopefully, this helps anyone who runs into this in the future.
I'm trying to change the permissions of multiple categories and channels using the following code:
1role = get(guild.roles, name="Fun")
2for channel in guild.channels:
3 overwrites = channel.overwrites_for(role)
4 overwrites.view_channel = False
5 await channel.set_permissions(role, overwrite=overwrites)
6
I get the following error:
1role = get(guild.roles, name="Fun")
2for channel in guild.channels:
3 overwrites = channel.overwrites_for(role)
4 overwrites.view_channel = False
5 await channel.set_permissions(role, overwrite=overwrites)
6await channel.set_permissions(role, overwrite=overwrites)
7discord.errors.Forbidden: 403 Forbidden (error code: 50013): Missing Permissions
8
It will work if I give the bot the administrator permission. However, it won't work if I give the bot every single other permission, the bot is at the top of the roles, and it has every category permission. What am I missing? I'm using Novus (fork of discord.py so it's pretty much the same thing).
Edit: I've asked people in the discord.py, Novus, and discord developers discord servers and they can't help me either. With the exception of the discord developers server they just didn't answer my question.
Edit 2: I was able to get it working when I manually added the bot's role to the category and enabled view_channel. This won't work, however, because that means the server owner has to manually do this for every category and channel which is really inconvenient.
ANSWER
Answered 2021-Oct-31 at 12:05This is simply how Discord works, if a channel is private and the bot does not have administrator permissions or role that has permissions to view the channel, it is not able to view it.
You can create a new category that the bot can access like this:
1role = get(guild.roles, name="Fun")
2for channel in guild.channels:
3 overwrites = channel.overwrites_for(role)
4 overwrites.view_channel = False
5 await channel.set_permissions(role, overwrite=overwrites)
6await channel.set_permissions(role, overwrite=overwrites)
7discord.errors.Forbidden: 403 Forbidden (error code: 50013): Missing Permissions
8role = await guild.create_role(name="Fun")
9category = await guild.create_category("category_name")
10await category.set_permissions(role, read_messages=True, send_messages=True, connect=True, speak=True)
11await category.set_permissions(ctx.guild.me, read_messages=True, send_messages=True, speak=True)
12
QUESTION
Error [ERR_REQUIRE_ESM]: require() of ES Module not supported
Asked 2022-Feb-03 at 22:08I'm trying to make a Discord bot that just says if someone is online on the game.
However I keep getting this message:
[ERR_REQUIRE_ESM]: require() of ES Module from not supported. Instead change the require of index.js in... to a dynamic import() which is available in all CommonJS modules.
This is my code:
1 module.exports = {
2 name: 'username',
3 description: "this is the username command",
4 async execute(message, args) {
5
6 const fetch = require('node-fetch');
7
8 if (args.length !== 1) {
9 return message.channel.send("invalid username wtf")
10 }
11
12 const ign = args[0]
13
14 if (ign.length > 16 || ign.length < 3) {
15 return message.channel.send("invalid username wtf")
16 }
17
18 const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
19 const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
20
21 if (uuid.length !== 32) {
22 return;
23 }
24
25 if (onlineinfo.success) {
26 if (onlineinfo.session.online) {
27 message.channel.send("they are online")
28 }
29 else {
30 message.channel.send("they are offline")
31 }
32 }
33 else {
34 message.channel.send("hypixel api bad wtf")
35 }
36 }
37 }
38
This is my package.json file:
1 module.exports = {
2 name: 'username',
3 description: "this is the username command",
4 async execute(message, args) {
5
6 const fetch = require('node-fetch');
7
8 if (args.length !== 1) {
9 return message.channel.send("invalid username wtf")
10 }
11
12 const ign = args[0]
13
14 if (ign.length > 16 || ign.length < 3) {
15 return message.channel.send("invalid username wtf")
16 }
17
18 const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
19 const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
20
21 if (uuid.length !== 32) {
22 return;
23 }
24
25 if (onlineinfo.success) {
26 if (onlineinfo.session.online) {
27 message.channel.send("they are online")
28 }
29 else {
30 message.channel.send("they are offline")
31 }
32 }
33 else {
34 message.channel.send("hypixel api bad wtf")
35 }
36 }
37 }
38{
39 "name": "discordbot",
40 "version": "1.0.0",
41 "main": "main.js",
42 "scripts": {
43 "test": "echo \"Error: no test specified\" && exit 1",
44 "start": "node main.js"
45 },
46 "author": "",
47 "license": "ISC",
48 "description": "",
49 "dependencies": {
50 "discord.js": "^13.0.1",
51 "node-fetch": "^3.0.0"
52 }
53}
54
ANSWER
Answered 2021-Sep-07 at 06:38node-fetch
v3 recently stopped support for the require
way of importing it in favor of ES Modules. You'll need to use ESM imports now, like:
1 module.exports = {
2 name: 'username',
3 description: "this is the username command",
4 async execute(message, args) {
5
6 const fetch = require('node-fetch');
7
8 if (args.length !== 1) {
9 return message.channel.send("invalid username wtf")
10 }
11
12 const ign = args[0]
13
14 if (ign.length > 16 || ign.length < 3) {
15 return message.channel.send("invalid username wtf")
16 }
17
18 const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
19 const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
20
21 if (uuid.length !== 32) {
22 return;
23 }
24
25 if (onlineinfo.success) {
26 if (onlineinfo.session.online) {
27 message.channel.send("they are online")
28 }
29 else {
30 message.channel.send("they are offline")
31 }
32 }
33 else {
34 message.channel.send("hypixel api bad wtf")
35 }
36 }
37 }
38{
39 "name": "discordbot",
40 "version": "1.0.0",
41 "main": "main.js",
42 "scripts": {
43 "test": "echo \"Error: no test specified\" && exit 1",
44 "start": "node main.js"
45 },
46 "author": "",
47 "license": "ISC",
48 "description": "",
49 "dependencies": {
50 "discord.js": "^13.0.1",
51 "node-fetch": "^3.0.0"
52 }
53}
54import fetch from "node-fetch";
55
at the top of your file.
QUESTION
How to check if a bot can DM a user
Asked 2022-Jan-22 at 22:03If a user has the privacy setting "Allow direct messages from server members" turned off and a discord bot calls
1await user.dm_channel.send("Hello there")
2
You'll get this error:
1await user.dm_channel.send("Hello there")
2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
3
I would like to check whether I can message a user without sending them a message. Trying to send a message and catching this error does not work for me, because I don't want a message to get sent in the event that the bot is allowed to message.
I have tried this:
1await user.dm_channel.send("Hello there")
2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
3print(user.dm_channel.permissions_for(bot).send_messages)
4
but it always returns True, even if the message is not permitted.
I have also tried this:
1await user.dm_channel.send("Hello there")
2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
3print(user.dm_channel.permissions_for(bot).send_messages)
4channel = await user.create_dm()
5if channel is None:
6 ...
7
but unfortunately, it seems that "has permission to message user" and "has permission to create a dm channel" are considered different.
EDIT
To clarify the exact usage since there seems to be a bit of confusion, take this example. There is a server, and 3 users in question: Me, My Bot, and Steve. Steve has "Allow direct messages from server members" checked off.
The bot has a command called !newgame
which accepts a list of users and starts a game amongst them, which involves DMing some of the members of the game. Because of Steve's privacy settings, he cannot play the game (since the bot will need to message him). If I do
1await user.dm_channel.send("Hello there")
2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
3print(user.dm_channel.permissions_for(bot).send_messages)
4channel = await user.create_dm()
5if channel is None:
6 ...
7!newgame @DJMcMayhem @Steve
8
I'd like to provide a response like:
1await user.dm_channel.send("Hello there")
2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
3print(user.dm_channel.permissions_for(bot).send_messages)
4channel = await user.create_dm()
5if channel is None:
6 ...
7!newgame @DJMcMayhem @Steve
8> I can't start a game with that list of users because @Steve has the wrong privacy settings.
9
But as far as I know right now, the only way to find out if Steve can play is by first attempting to message every user, which I'd like to avoid.
ANSWER
Answered 2022-Jan-22 at 22:03You can generate a Bad Request to the dm_channel. This can be accomplished by setting content to None, for example.
If it returns with 400 Bad Request
, you can DM them. If it returns with 403 Forbidden
, you can't.
1await user.dm_channel.send("Hello there")
2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
3print(user.dm_channel.permissions_for(bot).send_messages)
4channel = await user.create_dm()
5if channel is None:
6 ...
7!newgame @DJMcMayhem @Steve
8> I can't start a game with that list of users because @Steve has the wrong privacy settings.
9async def can_dm_user(user: discord.User) -> bool:
10 ch = user.dm_channel
11 if ch is None:
12 ch = await user.create_dm()
13
14 try:
15 await ch.send()
16 except discord.Forbidden:
17 return False
18 except discord.HTTPException:
19 return True
20
21
QUESTION
How can I send Dynamic website content to scrapy with the html content generated by selenium browser?
Asked 2022-Jan-20 at 15:35I am working on certain stock-related projects where I have had a task to scrape all data on a daily basis for the last 5 years. i.e from 2016 to date. I particularly thought of using selenium because I can use crawler and bot to scrape the data based on the date. So I used the use of button click with selenium and now I want the same data that is displayed by the selenium browser to be fed by scrappy. This is the website I am working on right now. I have written the following code inside scrappy spider.
1class FloorSheetSpider(scrapy.Spider):
2 name = "nepse"
3
4 def start_requests(self):
5
6 driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
7
8
9 floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
10
11 for date in floorsheet_dates:
12 driver.get(
13 "https://merolagani.com/Floorsheet.aspx")
14
15 driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
16 ).send_keys(date)
17 driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
18 total_length = driver.find_element(By.XPATH,
19 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
20 z = int((total_length.split()[-1]).replace(']', ''))
21 for data in range(z, z + 1):
22 driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
23 self.url = driver.page_source
24 yield Request(url=self.url, callback=self.parse)
25
26
27 def parse(self, response, **kwargs):
28 for value in response.xpath('//tbody/tr'):
29 print(value.css('td::text').extract()[1])
30 print("ok"*200)
31
Update: Error after answer is
1class FloorSheetSpider(scrapy.Spider):
2 name = "nepse"
3
4 def start_requests(self):
5
6 driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
7
8
9 floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
10
11 for date in floorsheet_dates:
12 driver.get(
13 "https://merolagani.com/Floorsheet.aspx")
14
15 driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
16 ).send_keys(date)
17 driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
18 total_length = driver.find_element(By.XPATH,
19 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
20 z = int((total_length.split()[-1]).replace(']', ''))
21 for data in range(z, z + 1):
22 driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
23 self.url = driver.page_source
24 yield Request(url=self.url, callback=self.parse)
25
26
27 def parse(self, response, **kwargs):
28 for value in response.xpath('//tbody/tr'):
29 print(value.css('td::text').extract()[1])
30 print("ok"*200)
312022-01-14 14:11:36 [twisted] CRITICAL:
32Traceback (most recent call last):
33 File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
34 result = current_context.run(gen.send, result)
35 File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
36 start_requests = iter(self.spider.start_requests())
37TypeError: 'NoneType' object is not iterable
38
I want to send current web html content to scrapy feeder but I am getting unusal error for past 2 days any help or suggestions will be very much appreciated.
ANSWER
Answered 2022-Jan-14 at 09:30The 2 solutions are not very different. Solution #2 fits better to your question, but choose whatever you prefer.
Solution 1 - create a response with the html's body from the driver and scraping it right away (you can also pass it as an argument to a function):
1class FloorSheetSpider(scrapy.Spider):
2 name = "nepse"
3
4 def start_requests(self):
5
6 driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
7
8
9 floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
10
11 for date in floorsheet_dates:
12 driver.get(
13 "https://merolagani.com/Floorsheet.aspx")
14
15 driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
16 ).send_keys(date)
17 driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
18 total_length = driver.find_element(By.XPATH,
19 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
20 z = int((total_length.split()[-1]).replace(']', ''))
21 for data in range(z, z + 1):
22 driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
23 self.url = driver.page_source
24 yield Request(url=self.url, callback=self.parse)
25
26
27 def parse(self, response, **kwargs):
28 for value in response.xpath('//tbody/tr'):
29 print(value.css('td::text').extract()[1])
30 print("ok"*200)
312022-01-14 14:11:36 [twisted] CRITICAL:
32Traceback (most recent call last):
33 File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
34 result = current_context.run(gen.send, result)
35 File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
36 start_requests = iter(self.spider.start_requests())
37TypeError: 'NoneType' object is not iterable
38import scrapy
39from selenium import webdriver
40from selenium.webdriver.common.by import By
41from scrapy.http import HtmlResponse
42
43
44class FloorSheetSpider(scrapy.Spider):
45 name = "nepse"
46
47 def start_requests(self):
48
49 # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
50 driver = webdriver.Chrome()
51
52 floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
53
54 for date in floorsheet_dates:
55 driver.get(
56 "https://merolagani.com/Floorsheet.aspx")
57
58 driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
59 ).send_keys(date)
60 driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
61 total_length = driver.find_element(By.XPATH,
62 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
63 z = int((total_length.split()[-1]).replace(']', ''))
64 for data in range(1, z + 1):
65 driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
66 self.body = driver.page_source
67
68 response = HtmlResponse(url=driver.current_url, body=self.body, encoding='utf-8')
69 for value in response.xpath('//tbody/tr'):
70 print(value.css('td::text').extract()[1])
71 print("ok"*200)
72
73 # return an empty requests list
74 return []
75
Solution 2 - with super simple downloader middleware:
(You might have a delay here in parse
method so be patient).
1class FloorSheetSpider(scrapy.Spider):
2 name = "nepse"
3
4 def start_requests(self):
5
6 driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
7
8
9 floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
10
11 for date in floorsheet_dates:
12 driver.get(
13 "https://merolagani.com/Floorsheet.aspx")
14
15 driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
16 ).send_keys(date)
17 driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
18 total_length = driver.find_element(By.XPATH,
19 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
20 z = int((total_length.split()[-1]).replace(']', ''))
21 for data in range(z, z + 1):
22 driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
23 self.url = driver.page_source
24 yield Request(url=self.url, callback=self.parse)
25
26
27 def parse(self, response, **kwargs):
28 for value in response.xpath('//tbody/tr'):
29 print(value.css('td::text').extract()[1])
30 print("ok"*200)
312022-01-14 14:11:36 [twisted] CRITICAL:
32Traceback (most recent call last):
33 File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
34 result = current_context.run(gen.send, result)
35 File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
36 start_requests = iter(self.spider.start_requests())
37TypeError: 'NoneType' object is not iterable
38import scrapy
39from selenium import webdriver
40from selenium.webdriver.common.by import By
41from scrapy.http import HtmlResponse
42
43
44class FloorSheetSpider(scrapy.Spider):
45 name = "nepse"
46
47 def start_requests(self):
48
49 # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
50 driver = webdriver.Chrome()
51
52 floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
53
54 for date in floorsheet_dates:
55 driver.get(
56 "https://merolagani.com/Floorsheet.aspx")
57
58 driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
59 ).send_keys(date)
60 driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
61 total_length = driver.find_element(By.XPATH,
62 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
63 z = int((total_length.split()[-1]).replace(']', ''))
64 for data in range(1, z + 1):
65 driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
66 self.body = driver.page_source
67
68 response = HtmlResponse(url=driver.current_url, body=self.body, encoding='utf-8')
69 for value in response.xpath('//tbody/tr'):
70 print(value.css('td::text').extract()[1])
71 print("ok"*200)
72
73 # return an empty requests list
74 return []
75import scrapy
76from scrapy import Request
77from scrapy.http import HtmlResponse
78from selenium import webdriver
79from selenium.webdriver.common.by import By
80
81
82class SeleniumMiddleware(object):
83 def process_request(self, request, spider):
84 url = spider.driver.current_url
85 body = spider.driver.page_source
86 return HtmlResponse(url=url, body=body, encoding='utf-8', request=request)
87
88
89class FloorSheetSpider(scrapy.Spider):
90 name = "nepse"
91
92 custom_settings = {
93 'DOWNLOADER_MIDDLEWARES': {
94 'tempbuffer.spiders.yetanotherspider.SeleniumMiddleware': 543,
95 # 'projects_name.path.to.your.pipeline': 543
96 }
97 }
98 driver = webdriver.Chrome()
99
100 def start_requests(self):
101
102 # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
103
104
105 floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
106
107 for date in floorsheet_dates:
108 self.driver.get(
109 "https://merolagani.com/Floorsheet.aspx")
110
111 self.driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
112 ).send_keys(date)
113 self.driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
114 total_length = self.driver.find_element(By.XPATH,
115 "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
116 z = int((total_length.split()[-1]).replace(']', ''))
117 for data in range(1, z + 1):
118 self.driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
119 self.body = self.driver.page_source
120 self.url = self.driver.current_url
121
122 yield Request(url=self.url, callback=self.parse, dont_filter=True)
123
124 def parse(self, response, **kwargs):
125 print('test ok')
126 for value in response.xpath('//tbody/tr'):
127 print(value.css('td::text').extract()[1])
128 print("ok"*200)
129
Notice that I've used chrome so change it back to firefox like in your original code.
QUESTION
Python/Selenium web scrap how to find hidden src value from a links?
Asked 2022-Jan-16 at 02:28Scrapping links should be a simple feat, usually just grabbing the src
value of the a tag.
I recently came across this website (https://sunteccity.com.sg/promotions) where the href value of a tags of each item cannot be found, but the redirection still works. I'm trying to figure out a way to grab the items and their corresponding links. My typical python selenium code looks something as such
1all_items = bot.find_elements_by_class_name('thumb-img')
2for promo in all_items:
3 a = promo.find_elements_by_tag_name("a")
4 print("a[0]: ", a[0].get_attribute("href"))
5
However, I can't seem to retrieve any href
, onclick
attributes, and I'm wondering if this is even possible. I noticed that I couldn't do a right-click, open link in new tab as well.
Are there any ways around getting the links of all these items?
Edit: Are there any ways to retrieve all the links of the items on the pages?
i.e.
1all_items = bot.find_elements_by_class_name('thumb-img')
2for promo in all_items:
3 a = promo.find_elements_by_tag_name("a")
4 print("a[0]: ", a[0].get_attribute("href"))
5https://sunteccity.com.sg/promotions/724
6https://sunteccity.com.sg/promotions/731
7https://sunteccity.com.sg/promotions/751
8https://sunteccity.com.sg/promotions/752
9https://sunteccity.com.sg/promotions/754
10https://sunteccity.com.sg/promotions/280
11...
12
ANSWER
Answered 2022-Jan-15 at 19:47You are using a wrong locator. It brings you a lot of irrelevant elements.
Instead of find_elements_by_class_name('thumb-img')
please try find_elements_by_css_selector('.collections-page .thumb-img')
so your code will be
1all_items = bot.find_elements_by_class_name('thumb-img')
2for promo in all_items:
3 a = promo.find_elements_by_tag_name("a")
4 print("a[0]: ", a[0].get_attribute("href"))
5https://sunteccity.com.sg/promotions/724
6https://sunteccity.com.sg/promotions/731
7https://sunteccity.com.sg/promotions/751
8https://sunteccity.com.sg/promotions/752
9https://sunteccity.com.sg/promotions/754
10https://sunteccity.com.sg/promotions/280
11...
12all_items = bot.find_elements_by_css_selector('.collections-page .thumb-img')
13for promo in all_items:
14 a = promo.find_elements_by_tag_name("a")
15 print("a[0]: ", a[0].get_attribute("href"))
16
You can also get the desired links directly by .collections-page .thumb-img a
locator so that your code could be:
1all_items = bot.find_elements_by_class_name('thumb-img')
2for promo in all_items:
3 a = promo.find_elements_by_tag_name("a")
4 print("a[0]: ", a[0].get_attribute("href"))
5https://sunteccity.com.sg/promotions/724
6https://sunteccity.com.sg/promotions/731
7https://sunteccity.com.sg/promotions/751
8https://sunteccity.com.sg/promotions/752
9https://sunteccity.com.sg/promotions/754
10https://sunteccity.com.sg/promotions/280
11...
12all_items = bot.find_elements_by_css_selector('.collections-page .thumb-img')
13for promo in all_items:
14 a = promo.find_elements_by_tag_name("a")
15 print("a[0]: ", a[0].get_attribute("href"))
16links = bot.find_elements_by_css_selector('.collections-page .thumb-img a')
17for link in links:
18 print(link.get_attribute("href"))
19
Community Discussions contain sources that include Stack Exchange Network