Explore all Bot open source software, libraries, packages, source code, cloud functions and APIs.

Popular New Releases in Bot

Telegram

Update to 8.7.1 (2629)

discord.js

@discordjs/voice@0.9.0

YYeTsBot

v2021-07-24

botkit

v4.15.0

beets

beets 1.6.0

Popular Libraries in Bot

Telegram

by DrKLO doticonjavadoticon

star image 19712 doticonGPL-2.0

Telegram for Android source

discord.js

by discordjs doticonjavascriptdoticon

star image 18961 doticonApache-2.0

A powerful JavaScript library for interacting with the Discord API

hubot

by hubotio doticonjavascriptdoticon

star image 15903 doticonMIT

A customizable life embetterment robot.

InstaPy

by timgrossmann doticonpythondoticon

star image 13182 doticonGPL-3.0

📷 Instagram Bot - Tool for automated Instagram interactions

YYeTsBot

by tgbot-collection doticonpythondoticon

star image 10806 doticonMIT

🎬 人人影视bot,完全对接人人影视全部无删减资源

botkit

by howdyai doticontypescriptdoticon

star image 10694 doticonMIT

Botkit is an open source developer tool for building chat bots, apps and custom integrations for major messaging platforms.

beets

by beetbox doticonpythondoticon

star image 10638 doticonMIT

music library manager and MusicBrainz tagger

awesome-quant

by wilsonfreitas doticonpythondoticon

star image 7607 doticon

A curated list of insanely awesome libraries, packages and resources for Quants (Quantitative Finance)

EZFN-Lobbybot

by EZFNDEV doticonpythondoticon

star image 6876 doticon

With EasyFNBot you can easily create you own Fortnite Lobby Bot in less than 5 minutes which will be online forever!

Trending New libraries in Bot

streetmerchant

by jef doticontypescriptdoticon

star image 4794 doticonMIT

🤖 The world's easiest, most powerful stock checker

Baileys

by adiwajshing doticonjavascriptdoticon

star image 2574 doticon

Lightweight full-featured WhatsApp Web + Multi-Device API

fairgame

by Hari-Nagarajan doticonpythondoticon

star image 2197 doticonGPL-3.0

Tool to help us buy hard to find items.

Discord-MusicBot

by SudhanPlayz doticonjavascriptdoticon

star image 1586 doticonNOASSERTION

An advanced discord music bot, supports Spotify, Soundcloud, YouTube with Shuffling, Volume Control and Web Dashboard with Slash Commands support!

thetagang

by brndnmtthws doticonpythondoticon

star image 1443 doticonNOASSERTION

ThetaGang is an IBKR bot for collecting money

Ultroid

by TeamUltroid doticonpythondoticon

star image 1355 doticonAGPL-3.0

Advanced Multi-Featured Telegram UserBot, Built in Python Using Telethon lib.

disc-11

by zhycorp doticontypescriptdoticon

star image 1223 doticonBSD-3-Clause

A dedicated open-source Discord bot for Zhycorp based on our Discord bot template with more features. Easy to use, and with no coding required.

oicq

by takayama-lily doticontypescriptdoticon

star image 1214 doticonMPL-2.0

QQ机器人协议库

NostalgiaForInfinity

by iterativv doticonpythondoticon

star image 1132 doticonGPL-3.0

Trading strategy for the Freqtrade crypto bot

Top Authors in Bot

1

Androz2091

24 Libraries

star icon1718

2

microsoft

22 Libraries

star icon18539

3

backmeupplz

19 Libraries

star icon1389

4

Koolwiza

14 Libraries

star icon130

5

adidoank

14 Libraries

star icon123

6

Tomato6966

13 Libraries

star icon489

7

ImJanindu

13 Libraries

star icon111

8

navaneethkm004

12 Libraries

star icon163

9

slack-ruby

12 Libraries

star icon1344

10

the-programmers-hangout

10 Libraries

star icon54

1

24 Libraries

star icon1718

2

22 Libraries

star icon18539

3

19 Libraries

star icon1389

4

14 Libraries

star icon130

5

14 Libraries

star icon123

6

13 Libraries

star icon489

7

13 Libraries

star icon111

8

12 Libraries

star icon163

9

12 Libraries

star icon1344

10

10 Libraries

star icon54

Trending Kits in Bot

ChatGPT is transforming all fields, including application development. You can easily add ChatGPT to your applications to provide additional intelligence.


Here are some cool open source libraries that you can leverage to add ChatGPT capabilities in your applications.


Start with Andrej Karpathy’s NanoGPT to get a feel for how GPT works at a smaller scale, even in your laptop.


ChatGPT Prompts is a great repo to help you leverage ChatGPT personas such as a translator, interviewer, JavaScript console, UI developer, IT architect, Excel sheet, stand-up comedian, philosopher, plagiarism checker, doctor, financial analyst, and many more.

If you'd like to use ChatGPT on your desktop try ChatGPT by lencx.


Up your chat game with ChatGPT apps for WhatApp, Telegram and Discord.


If you would like to use ChatGPT to compare with Google results from your browser, or integrate ChatGPT into your IDE or a general-purpose API to integrate with any other application like Telegram and others to create your custom ChatGPT integrations, below are useful repositories. Try them to build and get familiar with ChatGPT use cases.


You can build an intelligent telegram bot. It can automatically or on request send text, video, images, documents and perform different activities.   


Using a Python Telegram Bot Library, you can Broadcast, Teach, Collect Leads, Search, Reply, Remind, Play, Connect, etc. To help developers, these bots act as a BotFather. Also, you can implement the pair-to-pair security protocol and end-to-end encryption to ensure that each exchange of messages between the Bot and the user is secure. 


We have handpicked the top and trending Python Telegram Bot libraries for your next project below.  

python-telegram-bot

  • Used for building a bot in telegram easily by coupling with Python libraries like Flask, Requests, and Viz.  
  • Provides an asynchronous interface for the Telegram Bot API.  
  • The library features many high-level classes to make the development straightforward.  

Telethon

  • An asynchronous Python 3 MTProto library that helps interact with Telegram’s API as a user through a bot API alternative.  
  • If we install cryptg, the library will work faster as the encryption and decryption will be done using C instead of Python.  
  • If we install pillow, larger images will be automatically resized while sending photos to prevent telegram from failing with ‘Invalid Image’ messages.  

pyTelegramBotAPI

  • Used as the Python implementation for the telegram bot as it supports both synchronous and asynchronous methods.  
  • Provides functions like send_message, send_xyz, send_document, etc. And listens to all incoming messages.  
  • Can have an arbitrary name, but it should have only one parameter: the message.  

aiogram

  • Used for building bots quickly and efficiently using the available template and a fully asynchronous framework for Telegram Bot API.  
  • Used in Bot applications and Automation.  
  • Can reply into webhook, i.e., making requests in response to any latest updates.  

pyrogram

  • Modern, elegant, and asynchronous Telegram MTProto API framework in Python for bots and users.  
  • Enables you to easily interact with the main Telegram API using a user interface or a bot API alternative using Python.  
  • Types and methods are all type-hinted, which will enable excellent editor support.  

telepot

  • Helps build applications for Telegram Bot API and works on Python 2.7 and Python 3.  
  • Use telepot.glance() function for extracting the headline information.  
  • Supports synchronous and asynchronous methods of programming. 

mtprotoproxy

  • Fully asynchronous and can process a lot of connections.  
  • Not just a tool but has an API that can help customize the Telegram MTProto proxy.  
  • Can be used for logging, limiting access, and creating proxy farms that are hard to filter. 

BrainDamage

  • Used in Runtime environments, Docker applications, and Servers.  
  • Can destroy the active slaves, remove the stub from the host and registry entries, run shell commands on the host, and download files on a host computer.  
  • Used for Artificial Intelligence and Machine Learning.  

informer

  • Used in Docker applications, Bot, and Automation.  
  • Allows you to masquerade as multiple REAL users on telegram and spy on 500+ Telegram channels per account.  
  • Is a Telegram Mass Surveillance Bot in Python.  

TeleGram-Scraper

  • Used to export competitor groups, channel members, and add them to your own group or channel.  
  • You can scrape search results and extract the contents produced from those search results.  
  • Supports telegram premium API.  

telebot

  • Provides the best-of-its-kind API for command routing, keyboards, and inline query requests and callbacks. 
  • Are Concise API, supports Command routing, Middleware, Effortless bot callbacks, and Transparent File API.  
  • Is a highload-ready solution that has APIs that are easy to memorize and use.  

mautrix-telegram  

  • A hybrid Matrix- telegram puppeting or relaybot bridging library. 
  • Has 2-factor authentication enabled for logging in with a bot token.  
  • Includes a simple script to help migration between different database management systems. 

python-aria-mirror-bot

  • A telegram bot for mirroring files on the internet to our Google Drive or Telegram.  
  • Supports Mirroring direct download links to Google Drive, Upload and download progress, Docker support, Download or upload speed and ETAs, Index Link support, and many more.  
  • Stops duplicates for all tasks except for qBittorrent and youtube-dl tasks.  

TorrentLeech-Gdrive

  • Based on the Pyrogram library and Leecher.  
  • Supports Telegram file mirroring to the cloud with its unzipping, untar, and unrar.  
  • Help change the rclone destination configuration on the fly. 

FAQ:

1. What is the best API framework for building a Python Telegram bot?  

The Python-telegram-bot library is a fantastic framework for creating Python Telegram Bots. You can use it to create and manage Telegram bots with many features. This library makes it easier to use the Telegram Bot API by simplifying its complexity. It offers features like inline mode support, webhook integration, and message handling.  

  

2. How does the Telegram Bot API library compare to other API alternatives?  

Python-telegram-bot is a popular and well-maintained choice for making Telegram bots with Python. However, other libraries offer unique features and approaches. Here are some alternatives:  

  • python-telegram-bot  
  • pyTelegramBotAPI  
  • aiogram  
  • Telepot  

  

3. What are the main differences between the Telegram Bot API and the main Telegram API?  

Developers intend to use the Telegram Bot API to build custom clients. The platform provides special user experiences. The Bot API allows for automated bots to interact with users. The Telegram Bot API is complex. It requires a deeper understanding of the Telegram protocol. The Bot API is simpler and more user-friendly.  

The Telegram API is good for making different Telegram apps. The Bot API is for making bots that interact automatically. The Telegram API lets you have much control over messaging and interactions. The Bot API makes bot-specific tasks easier.  

The authentication process is different for the two APIs. To use the Telegram API, you need a user session. The Bot API, on the other hand, requires API tokens for authentication. The Telegram API includes messaging, customization, and file-sharing features. The Bot API is all about bot stuff, like commands, handling messages, and inline queries.  

  

4. How user-friendly is the interface of Python libraries for developing bots?  

Python libraries make bot development easier by providing user-friendly interfaces. But the level of user-friendliness can vary based on the specific library you choose. When evaluating how easy it is to use a library's interface, think about these things: 

Documentation:  

Developers need clear documentation with examples to use the library effectively.  

Ease of Setup:  

Installing and setting up a library can make developing easier and more efficient.  

Community Support:  

A strong community can offer assistance, answer queries, and share tips. It enhances the overall user experience.  

Abstraction:  

The extent to which the library abstracts complex API calls. It handles common use cases and can greatly impact usability.  

Flexibility:  

A library that is both powerful and easy to use is great for beginners and experts.  

  

5. Can one use Python as one of many programming languages to create a bot on Telegram?  

You can definitely use Python as a programming language to create a bot on Telegram. Telegram offers a Bot API. It offers a straightforward way to develop bots. Python has several libraries that make it easy to interact with this API. Some commonly used libraries include:  

python-telegram-bot:  

This is a popular library specifically designed to create Telegram bots in Python. The interface is easy to use and simplifies the Telegram Bot API. It makes it suitable for both beginners and experienced developers.  

aiogram:  

This is an asynchronous library. It uses Python's asyncio framework to build Telegram bots. It is particularly useful when handling many concurrent tasks in your bot.  

pyTelegramBot API:  

This library offers a simpler interface for basic bot functionalities. Beginners can easily create a basic bot without worrying about complicated details.  

Telepot:  

Telepot is an earlier library, but you can still use it to make Telegram bots with Python. However, it might not have newer libraries' latest features and updates.  

These libraries help interact with the Telegram Bot API using Python. You can use it to create bots that answer people, organize chats, and send messages.  

  

6. Can a Chat Bot or ChatGPT bot be built using Python libraries?  

It is possible to build a Chat Bot or ChatGPT bot using Python libraries. To create a chatbot, you can use Python. It doesn't have to be complex or need advanced techniques. Here are a few ways that can help you with the same:  

Rule-Based Chatbots:  

You can build a basic rule-based chatbot using nltk and spaCy. These provide tools for tokenization and part-of-speech. It can help analyze user input and generate responses based on predefined rules.  

Generative Chatbots:  

Creating a complete model can complicate things. But with the Open AI GPT-3 API, you can add a strong language model to your Python chatbot. OpenAI's API allows you to send prompts to GPT-3 and receive generated responses. It enables you to generate more human-like interactions. Using GPT-3 will need an API key and an understanding of API integration.  

Simple Retrieval-Based Chatbots:  

You can use Python to create a basic chatbot that gives set responses to input. You can use libraries like gensim and scikit-learn to create models. These models match user queries with responses. These models help determine text similarity.  

Custom Machine Learning Models:  

For more advanced chatbots, you can use libraries like transformers. It helps fine-tune pre-trained language models for specific chatbot tasks. This approach requires some familiarity with machine learning and natural language processing concepts.  

Bot Frameworks:  

Some bot frameworks use Python and have extra features for creating conversational AI. Rasa allows you to create rule-based and machine learning-based chatbots. It provides tools to train and deploy them.  

  

7. Are there any tutorials for programmers who want to make bots on Telegram with Python?  

There are tutorials for programmers who want to make bots on Telegram with Python. The tutorials explain how to create a bot, including setup and message handling. They also teach you how to add more advanced features. Here are a few resources to get you started:  

Python Telegram Bot Library Documentation:  

The official documentation for the 'python-telegram-bot' library provides a comprehensive guide. It helps build bots on Telegram using Python. It covers installation, message handling, inline queries, basic usage, and keyboard markup.  

Real Python Tutorial:  

Real Python has a tutorial. It guides you through building a weather bot for Telegram using Python. It covers using the 'python-telegram-bot' library and integrating with a weather API.  

YouTube Video Tutorials:  

YouTube hosts a wealth of video tutorials on building Telegram bots with Python. Search for keywords like 'Telegram Bot Python tutorial' to find relevant videos. It suits your learning style.  

FreeCodeCamp Tutorial:  

FreeCodeCamp offers a step-by-step tutorial on building a Telegram bot using Python. You will learn how to set up a bot, handle updates, create an interactive bot, and send messages.  

Dev.to Articles:  

You can find articles and tutorials about making Telegram bots with Python on Dev.to. These articles cover various topics, from basic bot creation to more advanced features.  

GitHub Repositories:  

Developers frequently share their Telegram bot projects on GitHub. They include the source code and documentation. Exploring these repositories can provide practical examples and insights.  

  

8. How difficult is learning Python to create a simple bot on Telegram?  

Creating a basic bot on Telegram using Python can be fairly easy to learn. The difficulty level depends on how much you know about programming and if you use tools. It also depends on your experience with Python. Here is a general overview:  

For Beginners:  

  • If you are new to programming, you may need time to learn Python's syntax and concepts.  
  • After learning Python, you can use it to explore the Telegram Bot API and Python libraries. 
  • We can make a bot that sends and gets messages, follows commands, and gives simple information. It can be achievable for beginners with some practice and patience.  

For experienced programmers:  

  • Knowing programming makes learning Python and the libraries for creating Telegram bots easy.  
  • It can be helpful to know about APIs and making HTTP requests. You'll use the Telegram Bot API to interact with Telegram.  
  • If you don't know a lot, you might need more time to learn about some things. These include inline searches, connecting with other apps, and managing messages.  

Beginner to intermediate programmers can create a basic bot on Telegram with Python. There are many helpful resources available to help you learn easily. These resources include documents, tutorials, and examples.  

  

9. When creating a telegram Bot with Python, are there any special things to consider compared to Java or C++?  

Creating a Telegram bot is similar in many programming languages. However, there are a few differences to think about. When making a Telegram bot, remember some important differences from other languages. 

  • Library Availability  
  • Development Speed  
  • Syntax and Conciseness  
  • Learning Curve  
  • Memory Management  
  • Asynchronous Vs. Synchronous  
  • Performance  
  • Integration with other Services  
  • Community and Libraries  
  • Deployment and Hosting  

Python is a language that has simplicity, strong community support, and user-friendly availability. This makes it a popular choice for developing Telegram bots. Consider familiarity, project requirements, and team expertise when choosing a language.  

  

10. Are there security concerns when using Python libraries to develop bots on Telegram?  

You should know about security issues if you create bots on Telegram or other platforms. Python is safe, but libraries, code, and architecture can affect your bot's security. Here are some common security considerations:  

  • Input Validation and Sanitization  
  • API Token Protection  
  • Rate Limiting and Anti-Abuse Measures  
  • Bot Permissions  
  • Webhook Security  
  • Secure Coding Practices  
  • Library Security and Updates  
  • Data Privacy and Storage  
  • User Privacy  
  • Monitoring and Logging  
  • Secure Deployment  

Build an end-to-end automated trading application to visualize financial data of stock holdings. Buy/sell stocks automatically using Python trading bot libs.

 

The key functions of a stock trading bot built using the Python algorithmic trading libraries are getting the data, training the model, entering and exiting trades, placing orders, and getting notified. Through backtesting on historical data and by minimizing human intervention, a Python trading bot can increase the speed of order entry and establish discipline in the volatile stock market to help traders earn profits. Since computer systems react quickly to fluctuating markets, an automated Python-based trading system can produce bids as soon as the defined trading conditions are met.

 

Browse below to check out the latest and trending Python libraries to build your automated trading bot:

zipline 

  • Backtesting and algorithm development framework for trading strategies. 
  • Easily integrates with live trading platforms. 
  • Provides a range of built-in statistical and risk analysis tools. 

ta-lib 

  • Offers over 150 technical analysis indicators. 
  • Widely used in financial markets for quantitative analysis. 
  • Supports multiple programming languages, including Python. 

freqtrade 

  • High-frequency trading bot with a focus on cryptocurrencies. 
  • Supports machine learning and AI-based strategy development. 
  • Offers an active community and a range of pre-built strategies.

binance-trade-bot 

  • Designed specifically for trading on the Binance exchange. 
  • Supports various trading pairs and strategies. 
  • Offers real-time market data and order execution. 

jesse 

  • Python-based framework for cryptocurrency trading strategies. 
  • Provides a simple and clean syntax for strategy development. 
  • Supports multiple exchanges and backtesting. 

Crypto-Signal 

  • Cryptocurrency trading signals and alert system. 
  • Offers technical analysis indicators and custom signal creation. 
  • Integration with popular cryptocurrency exchanges. 

Binance-volatility-trading-bot 

  • Designed for volatility-based trading strategies on Binance. 
  • Focuses on profiting from price fluctuations. 
  • Offers various trading parameters for customization. 

binance-trader 

  • A simple and lightweight Binance trading bot. 
  • Supports spot and futures trading on the Binance platform. 
  • Suitable for both beginners and experienced traders. 

OctoBot 

  • Modular and extensible cryptocurrency trading bot. 
  • Provides a web-based dashboard for monitoring and configuration. 
  • Supports various technical analysis indicators. 

cointrol 

  • Open-source cryptocurrency trading bot with a focus on Bitcoin. 
  • Supports multiple exchanges and customizable strategies. 
  • Offers risk management and trading performance analysis. 

Cryptocurrency-Trading-Bots-Python-Beginner-Advance 

  • A collection of Python trading bot examples for all skill levels. 
  • Educational resource with various trading strategies. 
  • Suitable for beginners and experienced traders. 

algobot 

  • Algorithmic trading bot framework with a range of features. 
  • Supports backtesting, live trading, and data analysis. 
  • Integrates with various data sources and brokers. 

binance-tutorials 

  • Educational resources and tutorials for Binance trading. 
  • Covers various aspects of trading on the Binance platform. 
  • Suitable for traders looking to learn and improve their skills. 

pybacktest 

  • A Python library for backtesting trading strategies. 
  • Supports historical price data analysis and strategy optimization. 
  • Provides visualizations for performance analysis. 

crypto-trader 

  • A simple cryptocurrency trading bot for Bitcoin. 
  • Designed for automated trading on various exchanges. 
  • Offers basic trading strategies. 

stock-trading-ml 

  • Machine learning-based stock trading bot. 
  • Uses historical stock data for predictive trading. 
  • Suitable for those interested in AI-driven trading. 

tradingview-webhooks-bot 

  • A bot that uses TradingView webhooks for automated trading. 
  • Integrates with TradingView alerts to trigger trades. 
  • Suitable for traders using TradingView for technical analysis. 

crypto-arbitrage 

  • Focuses on cryptocurrency arbitrage trading strategies. 
  • Aims to profit from price differences on different exchanges. 
  • Supports multiple cryptocurrency pairs. 

TensorFlow-Bitcoin-Robot 

  • Utilizes TensorFlow for machine learning-based Bitcoin trading. 
  • Provides tools for developing and training trading models. 
  • Suitable for those with machine learning expertise. 

pyjuque 

  • Cryptocurrency trading bot framework with a focus on flexibility. 
  • Supports custom strategies and risk management. 
  • Designed for traders with coding experience. 

Titan 

  • Algorithmic trading bot for various asset classes, including cryptocurrencies. 
  • Offers machine learning capabilities and strategy customization. 
  • Designed for both beginners and experienced traders. 

PyTrendFollow 

  • A trend-following trading bot framework. 
  • Focuses on capturing trends in various financial markets. 
  • Supports strategy development and live trading. 

Bitcoin-Surge-Trading-Alpha 

  • A cryptocurrency trading bot with a focus on capturing surges in Bitcoin prices. 
  • Designed for short-term trading strategies. 
  • Offers features for risk management and portfolio tracking.

pytrader 

  • A simple and lightweight cryptocurrency trading bot. 
  • Supports multiple exchanges and basic trading strategies. 
  • Suitable for beginners and traders looking for a minimalistic approach. 

FAQ   

1. How to choose a Python trading bot library for backtesting and live trading?   

When choosing a Python trading bot library, consider:- 

1. versatility in strategy development,  

2. access to historical and real-time data, and  

3. support for various order types.  

Seek libraries with intuitive APIs, risk controls, and exchange compatibility. Also, a vibrant community and regular updates are essential for long-term reliability.  

  

2. Can Python trading bot libraries handle multiple trading pairs?   

Python trading bot libraries are typically flexible and can handle multiple trading pairs. These libraries adapt to diverse markets, including cryptocurrencies, traditional stocks, and more. 

  

3. How do Python trading bot libraries handle risk management to protect investments?     

Risk management is paramount in Python trading bot libraries. Traders can apply stop-loss, position sizing, and risk controls to protect investments. Users can set preset loss limits, reducing the impact of market downturns. 

  

4. What tools do Python trading bot libraries provide for performance analysis?   

Python trading bot libraries provide tools for strategy optimization and performance analysis. Traders can backtest their strategies using historical data to fine-tune their algorithms. Performance metrics and visualization tools help assess strategy effectiveness and make informed adjustments.  

  

5. Can Python trading bot libraries run on cloud servers or require infrastructure?

Python trading bot libraries can be deployed on cloud servers, eliminating hardware needs. This cloud-based approach offers scalability, cost-efficiency, and accessibility for users. Thus making it an attractive option for both beginners and experienced traders. 

Here are some famous NodeJs Telegram Bot Libraries. Some of the use cases of NodeJs Telegram Bot Libraries include automating customer service, creating interactive experiences, automating workflow, and creating analytics dashboards.

 

Node.js Telegram Bot Libraries are libraries written in JavaScript that can be used to create custom Telegram bots. They provide a convenient way to write code that interacts with the Telegram Bot API, allowing developers to quickly create and deploy bots that can listen for commands, respond to messages, and perform other tasks.


Let us look at the libraries in detail below.

node-telegram-bot-api

  • Has built-in support for multiple programming languages.
  • Comprehensive API documentation.
  • Automated message queueing system.

nlp.js

  • Offers an advanced Natural Language Processing (NLP) engine.
  • Highly modular and customizable.
  • Built on the Node.js platform, so it is lightweight and fast.

telebot 

  • Provides an all-in-one solution for creating, managing, and deploying Telegram bots. 
  • Supports inline query and inline keyboard. 
  • Provides a unified API for all Telegram API endpoints. 

LiveBot 

  • Offers a powerful webhook system. 
  • Has a built-in analytics system. 
  • Allows developers to easily deploy and manage bots on popular hosting services. 

telegram-bot-api 

  • You do not need to know any programming language to get started. 
  • Makes it easier to build bots that can react quickly to incoming messages. 
  • Wide range of API methods for creating bots and interacting with users. 

telegram-client 

  • You can quickly get a bot up and running with minimal effort. 
  • With a built-in command processor. 
  • Used to send out mass messages or schedule messages to be sent at specific times.

teleapi 

  • Provides an easy way to create and manage multiple bots using a single interface. 
  • Offers an intuitive GUI for quickly setting up your bots and customizing their behavior. 
  • Lightweight and simple library that lets you quickly and easily build bots. 

BocchiBot 

  • Modern and extendable architecture. 
  • Extensive library of custom plugins. 
  • Integrated analytics system. 

instauto 

  • Includes an easy-to-use query builder. 
  • Provides an easy and secure authentication process for users. 
  • Provides unique methods such as the getStarted method which can be used to quickly set up the bot and get started. 

mtproto-core 

  • Supports the full MTProto protocol, which is a binary protocol developed by Telegram. 
  • Designed for both client and server sides, allowing developers to use the same library for both sides of the connection. 
  • Designed to be highly performant, allowing for fast and efficient communication between the bot and Telegram servers. 

Python Weather API Libraries offer an interface for accessing weather data from different APIs. Easy-to-use functions help fetch current weather, historical weather data, and weather forecasts. It supports different languages and units of measurement.  


Python Weather API Libraries can retrieve weather data for specific and multiple locations. It can retrieve information like humidity, precipitation, temperature, wind speed, and UV index. It offers customization options for specifying the level of granularity or detail of weather data required. It can handle exceptions or errors during the API request process. We can integrate with weather APIs like Dark Sky, Weatherbit, OpenWeatherMap, and AccuWeather.  


Here are the 7 best Python Weather API Libraries for helping developers: 

wttr.in: 

  • Is a weather service that offers weather data using a command-line interface.  
  • Allows you to retrieve current forecast and weather data for any location using a simple HTTP request.  
  • Offers various customization options like the language used for the data, the ability to specify the weather data format, and more.  
  • Supports various weather-related queries like sunsets, moon phases, and sunrises. 

Jarvis: 

  • Is an open source library that offers an easy-to-use interface to create a virtual assistant like the character Jarvis in Iron Man.  
  • Allows you to create custom commands and responses for the virtual assistant. 
  • Allow easy interaction with users and perform tasks.  
  • Support is offered for speech-to-text and text-to-speech capabilities.  
  • Offers the ability to integrate with third-party services and APIs, weather services, and Wikipedia. 

weewx: 

  • Is a Python library for interfacing with logging weather and weather station data.  
  • Offers a flexible framework to collect, display, and process weather data from various sources.  
  • Built-in support to upload data to social media and weather services platforms.  
  • Is an excellent choice for developers looking to create a weather station. 
  • Helps integrate weather data into applications. 

pysteps: 

  • Is an open source Python library for precipitation nowcasting.  
  • Offers various algorithms and tools for analysis, processing, and visualization.  
  • Offers precipitation data with a focus on high-resolution radar data.  
  • Is a powerful tool for meteorologists, researchers, and others working with precipitation data. 

wetterdienst: 

  • Is an open source library for accessing data from various German weather services.  
  • Offers a unified interface for accessing current, historical, and forecast weather data.  
  • Supports various weather data sources like Meteogroup, OpenWeatherMap, and Deutscher Wetterdienst.  
  • Offers an easy-to-use interface to access weather data. 

pirate-weather-ha: 

  • Is designed to replace the DarkSky integration in Home Assistant with a modified and updated version.  
  • The underlying code will be significantly different from its previous versions.  
  • The API key can be received from the Pirate Weather Site and used for tracking usage and keeping AWS bills reasonable. 

e_paper_weather_display: 

  • Is an open source library to display weather information on an e-paper display.  
  • Offers an easy-to-use interface for retrieving weather data from different sources. 
  • Helps display it on an e-paper display.  
  • Supports various weather data sources like Meteoblue and OpenWeatherMap.  
  • Is a valuable resource for anyone looking to build a weather display for commercial and personal use. 

Here are some of the famous Go Telegram Bot Libraries. These Libraries are used for Building custom bots for customer service, Developing bots for e-commerce and Building bots for entertainment.


Go Telegram Bot Libraries are libraries written in the programming language Go which provide an interface to the Telegram Bot API. They allow developers to create and interact with Telegram bots in their own applications. They provide a range of features such as easy setup, custom commands, webhooks, and more.


Let us have a look at these libraries in detail below.

Telegram

  • Provides a highly customizable environment to create bots.
  • Provides a Botstore, which is a library of bots created by other developers.
  • Provides excellent security for bots, with end-to-end encryption and access control.

telegram-bot-api

  • Provides webhooks and inline keyboards for interaction with users.
  • Supports deep linking to external websites and content within the Telegram app.
  • Offers a unique set of features such as group chats, channels, and sticker sets.

TelegramBots

  • Offers advanced features such as game playing, natural language processing, and automated responses.
  • Supports webhooks which allow you to receive messages and updates from other services and systems.
  • Allows you to customize your bots with a variety of useful features such as message templates

Telebot

  • Offers secure bot creation and hosting with built-in features to protect user data.
  • Designed to deliver reliable performance even when handling large volumes of requests.
  • Offers advanced features such as inline keyboards, custom commands and user profiles.

telegram-bot-sdk

  • Supports multiple programming languages, such as JavaScript, Python, and Java.
  • Included in Telegram-bot-sdk is well-documented and easy to understand.
  • Enables developers to quickly create and customize powerful bots to fit their needs.

go-tgbot

  • Provides a powerful command line tool that helps developers to quickly create and manage their Telegram bots.
  • Supports all the standard Telegram API features and adds several unique features such as custom keyboards, inline bot queries and more.
  • Supports a wide range of databases, including MongoDB, Redis, BoltDB and more.

telegram-bot-go

  • Quickly build and deploy a Telegram bot with minimal effort.
  • Comes with comprehensive documentation and tutorials.
  • Offers advanced features such as custom commands, inline commands, and support for multiple languages.

gotelebot

  • Provides a convenient way to create custom commands for your bot.
  • Supports webhooks, allowing you to create more interactive bots quickly and easily.
  • Support for all the major Telegram Bot API features and extras.

Here are some famous C# Telegram Bot Libraries. Some of the use cases of C# Telegram Bot Libraries include automated Customer Support, News Aggregator, Online Ordering, Scheduling Appointments. 


C# telegram bot libraries are collections of code that allow developers to create Telegram bots using the C# programming language. These libraries provide an API for developers to access the Telegram Bot API and create their own custom bots with a range of features. They also provide tools for creating interactive user experiences. 


Let us look at the libraries in detail below. 

Telegram.Bot 

  • Highly customizable and allows developers to tailor their bots for performing specific tasks. 
  • Provides advanced features such as message scheduling and inline keyboards. 
  • Built with security in mind, ensuring that your bots are secure and protected. 

TelegramBotClients 

  • Built-in support for the Telegram Bot API. 
  • Provide support for different types of bots. 
  • Highly extensible, allowing you to add your own custom features to your bot. 

NetTelegramBoApi 

  • Provides an easy way to integrate with third-party services. 
  • Supports Webhooks, allowing developers to easily receive updates from Telegram Bot API. 
  • Offers support for uploading and downloading files. 

pmcenter  

  • Support for dynamic message creation and message modification.  
  • Support for custom commands and automated responses. 
  • Advanced analytics to track conversations and user interactions.

CryptoGramBot 

  • Uses an asynchronous event-driven programming model to ensure that the bot responds quickly and efficiently to user interactions. 
  • Supports a wide range of languages, including C#, VB.NET, and F#. 
  • Provides built-in integration with Microsoft Azure and other cloud services. 

telegram-bot-groupagree 

  • Advanced chatbot features allow it to understand natural language input, leading to more accurate and efficient responses. 
  • Built-in support for payment systems, enabling easy monetization of bots. 
  • Uses machine learning algorithms to improve the accuracy of its responses. 

Telegram.Bot.Framework 

  • Allows advanced message routing. 
  • Contains a rich set of features such as inline queries, location tracking, and message editing. 
  • Provides secure authentication, allowing users to connect with your bot without having to share their login details. 

Telegram.BotAPI 

  • Provides an easy-to-use interface for developers to create bots. 
  • Supports asynchronous API calls, allowing developers to send multiple messages in one go. 
  • Allows developers to build bots that support a wide range of messaging capabilities. 

Telebot 

  • Only C# Telegram Bot library that supports .NET Standard 2.0, making it cross-platform compatible. 
  • Easy to integrate bot logic into existing applications. 
  • Provides a powerful and intuitive way to create custom commands and actions that can be triggered from anywhere within the bot. 

Twitter has long been seen as a short-text social platform. However, content creation on Twitter has scaled over the years across News, Questions, Announcements, Memes, Flash Sales, Teasers, Promos, Polls, Tips and others.


Elon Musk recently announced that Twitter would share ad revenue with content creators on the platform. This is a very interesting development and a step in the positive direction. How platforms monetize content and how the creators are compensated has been debated for a long time and are still evolving given the concentration of power with the platforms. Ad revenue sharing in a micro blogging site like Twitter can open up newer forms of journalism and other fast content opportunities.

The following are popular libraries for intelligent automation on the Twitter platform that can help with your content management, authoring, and engagement activities.


Consider the below popular open source libraries for integration with Twitter and Automation of tasks

Trending Discussions on Bot

How can you create a pop-up window in Discord that accepts an input from the user?

Python Selenium AWS Lambda Change WebGL Vendor/Renderer For Undetectable Headless Scraper

Is there a way to access the children of a CategoryChannel before it is deleted? [Discord.js]

Spring Boot WebClient stops sending requests

How do I get mobile status for discord bot by directly modifying IDENTIFY packet?

Changing Category/Channels Permissions Returns Error "Missing Permissions" - Novus/Discord.py

Error [ERR_REQUIRE_ESM]: require() of ES Module not supported

How to check if a bot can DM a user

How can I send Dynamic website content to scrapy with the html content generated by selenium browser?

Python/Selenium web scrap how to find hidden src value from a links?

QUESTION

How can you create a pop-up window in Discord that accepts an input from the user?

Asked 2022-Mar-30 at 07:14

It's my first time seeing this feature from a Discord bot. I tried looking everywhere but it seems that I have failed. There's this feature from Captcha.bot Discord bot where you can accept input from a pop-up window inside Discord.

There's a button in an embedded message made by Captcha.bot where you will have to answer a Captcha test. After pressing the button, it creates a pop-up window like this.

enter image description here

After placing the right answer on the captcha bot, here's the aftermath of the experience.

enter image description here

All I want to learn is how to summon that pop-up window using Discord.js if it's even possible or at least learn how they did it.

ANSWER

Answered 2022-Mar-30 at 07:12

Those are called modals, and they will be available in the next discord.js version, v14. There is already a pull request for this.

In the meantime, you can use an npm package like discord-modals or discordjs-modal.

You can find a working example with the discord-modals package below. Don't forget to install it first using npm i discord-modals.

1const {
2  Client,
3  Intents,
4  MessageActionRow,
5  MessageButton,
6} = require('discord.js');
7const discordModals = require('discord-modals');
8const { Modal, TextInputComponent, showModal } = discordModals;
9
10const TOKEN = 'YOUR TOKEN HERE';
11const client = new Client({
12  intents: [Intents.FLAGS.GUILDS, Intents.FLAGS.GUILD_MESSAGES],
13});
14discordModals(client);
15
16client.on('messageCreate', (message) => {
17  if (message.author.bot) return;
18
19  let button = new MessageActionRow();
20  button.addComponents(
21    new MessageButton()
22      .setCustomId('verification-button')
23      .setStyle('PRIMARY')
24      .setLabel('Open modal dialog'),
25  );
26  message.reply({
27    components: [button],
28  });
29});
30
31client.on('interactionCreate', async (interaction) => {
32  if (interaction.isButton()) {
33    if (interaction.customId === 'verification-button') {
34      const modal = new Modal() // We create a Modal
35        .setCustomId('verification-modal')
36        .setTitle('Verify yourself')
37        .addComponents([
38          new TextInputComponent()
39            .setCustomId('verification-input')
40            .setLabel('Answer')
41            .setStyle('SHORT')
42            .setMinLength(4)
43            .setMaxLength(12)
44            .setPlaceholder('ABCDEF')
45            .setRequired(true),
46        ]);
47
48      showModal(modal, {
49        client,
50        interaction,
51      });
52    }
53  }
54});
55
56client.on('modalSubmit', async (modal) => {
57  if (modal.customId === 'verification-modal') {
58    const response = modal.getTextInputValue('verification-input');
59    modal.reply(`Yay, your answer is submitted: "${response}"`);
60  }
61});
62
63client.once('ready', () => {
64  console.log('Bot v13 is connected...');
65});
66
67client.login(TOKEN);
68

enter image description here

Source https://stackoverflow.com/questions/71672321

QUESTION

Python Selenium AWS Lambda Change WebGL Vendor/Renderer For Undetectable Headless Scraper

Asked 2022-Mar-21 at 20:19
Concept:

Using AWS Lambda functions with Python and Selenium, I want to create a undetectable headless chrome scraper by passing a headless chrome test. I check the undetectability of my headless scraper by opening up the test and taking a screenshot. I ran this test on a Local IDE and on a Lambda server.


Implementation:

I will be using a python library called selenium-stealth and will follow their basic configuration:

1stealth(driver,
2        languages=["en-US", "en"],
3        vendor="Google Inc.",
4        platform="Win32",
5        webgl_vendor="Intel Inc.",
6        renderer="Intel Iris OpenGL Engine",
7        fix_hairline=True,
8        )
9

I implemented this configuration on a Local IDE as well as an AWS Lambda Server to compare the results.


Local IDE:

Found below are the test results running on a local IDE: enter image description here


Lambda Server:

When I run this on a Lambda server, both the WebGL Vendor and Renderer are blank. as shown below:

enter image description here

I even tried to manually change the WebGL Vendor/Renderer using the following JavaScript command:

1stealth(driver,
2        languages=["en-US", "en"],
3        vendor="Google Inc.",
4        platform="Win32",
5        webgl_vendor="Intel Inc.",
6        renderer="Intel Iris OpenGL Engine",
7        fix_hairline=True,
8        )
9driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})
10

Then I thought maybe that it could be something wrong with the parameter number. I configured the command execution without the if statement, but the same thing happened: It worked on my Local IDE but had no effect on an AWS Lambda Server.

Simply Put:

Is it possible to add Vendor/Renderer on AWS Lambda? In my efforts, it seems that there is no possible way. I made sure to submit this issue on the selenium-stealth GitHub Repository.

ANSWER

Answered 2021-Dec-18 at 02:01
WebGL

WebGL is a cross-platform, open web standard for a low-level 3D graphics API based on OpenGL ES, exposed to ECMAScript via the HTML5 Canvas element. WebGL at it's core is a Shader-based API using GLSL, with constructs that are semantically similar to those of the underlying OpenGL ES API. It follows the OpenGL ES specification, with some exceptions for the out of memory-managed languages such as JavaScript. WebGL 1.0 exposes the OpenGL ES 2.0 feature set; WebGL 2.0 exposes the OpenGL ES 3.0 API.

Now, with the availability of Selenium Stealth building of Undetectable Scraper using Selenium driven ChromeDriver initiated Browsing Context have become much more easier.


selenium-stealth

selenium-stealth is a python package selenium-stealth to prevent detection. This programme tries to make python selenium more stealthy. However, as of now selenium-stealth only support Selenium Chrome.

  • Code Block:

1stealth(driver,
2        languages=["en-US", "en"],
3        vendor="Google Inc.",
4        platform="Win32",
5        webgl_vendor="Intel Inc.",
6        renderer="Intel Iris OpenGL Engine",
7        fix_hairline=True,
8        )
9driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})
10from selenium import webdriver
11from selenium.webdriver.chrome.options import Options
12from selenium.webdriver.chrome.service import Service
13from selenium_stealth import stealth
14
15options = Options()
16options.add_argument("start-maximized")
17options.add_experimental_option("excludeSwitches", ["enable-automation"])
18options.add_experimental_option('useAutomationExtension', False)
19s = Service('C:\\BrowserDrivers\\chromedriver.exe')
20driver = webdriver.Chrome(service=s, options=options)
21
22# Selenium Stealth settings
23stealth(driver,
24      languages=["en-US", "en"],
25      vendor="Google Inc.",
26      platform="Win32",
27      webgl_vendor="Intel Inc.",
28      renderer="Intel Iris OpenGL Engine",
29      fix_hairline=True,
30  )
31
32driver.get("https://bot.sannysoft.com/")
33
  • Browser Screenshot:

  • bot_sannysoft

    You can find a detailed relevant discussion in Can a website detect when you are using Selenium with chromedriver?


    Changing WebGL Vendor/Renderer in AWS Lambda

    AWS Lambda enables us to deliver compressed WebGL websites to end users. When requested webpage objects are compressed, the transfer size is reduced, leading to faster downloads, lower cloud storage fees, and lower data transfer fees. Improved load times also directly influence the viewer experience and retention, which helps in improving website conversion and discoverability. Using WebGL, websites are more immersive while still being accessible via a browser URL. Through this technique AWS Lambda to automatically compress the objects uploaded to S3.

    product-page-diagram_Lambda-RealTimeFileProcessing.a59577de4b6471674a540b878b0b684e0249a18c

    Background on compression and WebGL

    HTTP compression is a capability that can be built into web servers and web clients to improve transfer speed and bandwidth utilization. This capability is negotiated between the server and the client using an HTTP header which may indicate that a resource being transferred, cached, or otherwise referenced is compressed. AWS Lambda on the server-side supports Content-Encoding header.

    On the client-side, most browsers today support brotli and gzip compression through HTTP headers (Accept-Encoding: deflate, br, gzip) and can handle server response headers. This means browsers will automatically download and decompress content from a web server at the client-side, before rendering webpages to the viewer.


    Conclusion

    Due to this constraint you may not be able to change the WebGL Vendor/Renderer in AWS Lambda, else it may directly affect the process of rendering webpages to the viewers and can stand out to be a bottleneck in UX.


    tl; dr

    You can find a couple of relevant detailed discussion in:

    Source https://stackoverflow.com/questions/70265306

    QUESTION

    Is there a way to access the children of a CategoryChannel before it is deleted? [Discord.js]

    Asked 2022-Feb-19 at 14:09

    I handle a channelDelete event in my discord bot. My original intent was to do the following:

    1. Listen for when a channel is deleted
    2. Check to see if its type equals 'GUILD_CATEGORY'
    3. Delete all the channels under that category

    I can typically access channels under a CategoryChannel through its property called children anywhere else except during this event...

    1module.exports = {
    2    name: 'channelDelete',
    3    once: false,
    4    async execute(channel) {
    5        if(channel.type == 'GUILD_CATEGORY')
    6            for(let [child_id, child] of channel.children)
    7                if (child.deletable) child.delete();
    8    },
    9}
    10

    I can confirm the code is being executed, but the problem is that the incoming channel object is in a state where it is already deleted, and I cannot get the children:

    • During debugging, I noticed the channel object has the property deleted: true
    • The children property is empty, even though I know there were channels in that channel category prior to deletion

    Is there a way for my bot to collect and handle the children of a CategoryChannel prior to its deletion?

    ANSWER

    Answered 2022-Feb-19 at 14:09
    Why?

    Unfortunately, this is how CategoryChannels work in discord.js...
    When the category is deleted, discord.js sends a request to the API to delete the channel. Only then, Discord sends you the event after the category is deleted.
    What happens next is that the children are not located in the category anymore! So you will not be able to get the children inside the CategoryChannel object.


    This is the code for the children property

    1module.exports = {
    2    name: 'channelDelete',
    3    once: false,
    4    async execute(channel) {
    5        if(channel.type == 'GUILD_CATEGORY')
    6            for(let [child_id, child] of channel.children)
    7                if (child.deletable) child.delete();
    8    },
    9}
    10get children() {
    11    return this.guild.channels.cache.filter(c => c.parentId === this.id);
    12}
    13

    It filters the channels which are in the guild, but those children channels are not located in the category anymore. Which is the reason the property is empty.

    The (Only?) Solution

    This will require caching the channels themselves. As there aren't really any other solutions.
    You may cache the channels in different ways.
    Just remember... there are instances and references in Javascript, and failure to acknowledge may lead to weird behaviors. The following code only works for small bots without sharding, just so you know.

    1module.exports = {
    2    name: 'channelDelete',
    3    once: false,
    4    async execute(channel) {
    5        if(channel.type == 'GUILD_CATEGORY')
    6            for(let [child_id, child] of channel.children)
    7                if (child.deletable) child.delete();
    8    },
    9}
    10get children() {
    11    return this.guild.channels.cache.filter(c => c.parentId === this.id);
    12}
    13const cachedChannels = new Map()
    14
    15client.on('ready', () => {
    16  // Looping thru guilds...
    17  client.guilds.forEach((guild) => {
    18    // Filtering the channels to category channels only and looping thru them...
    19    guild.channels.filter((channel) => channel.type === "GUILD_CATEGORY").forEach((category) => {
    20      // Note on references and instances: Numbers, Strings and Booleans do not have instances like Object, Arrays, etc. do
    21      // We are using an array here to abuse the references
    22      cachedChannels.set(category.id, [])
    23
    24      // Looping thru the children channels...
    25      category.children.forEach((children) => {
    26        // This is the array we've created earlier
    27        const storedChannels = cachedChannels.get(category.id)
    28
    29        // Pushing the ID so we can fetch them later
    30        storedChannels.push(children.id)
    31      })
    32    })
    33  })
    34})
    35
    36client.on('channelDelete', (channel) => {
    37  if (channel.type === "GUILD_CATEGORY") {
    38    // Looping thru our cached channel IDs defined earlier
    39    cachedChannels.get(channel.id).forEach((id) => {
    40      const child = channel.guild.channels.get(id)
    41      child.delete()
    42    })
    43
    44    // Delete the cached IDs to save memory
    45    cachedChannels.delete(channel.id)
    46  }
    47})
    48
    49

    Source https://stackoverflow.com/questions/70563152

    QUESTION

    Spring Boot WebClient stops sending requests

    Asked 2022-Feb-18 at 14:42

    I am running a Spring Boot app that uses WebClient for both non-blocking and blocking HTTP requests. After the app has run for some time, all outgoing HTTP requests seem to get stuck.

    WebClient is used to send requests to multiple hosts, but as an example, here is how it is initialized and used to send requests to Telegram:

    WebClientConfig:

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8

    The same ReactorClientHttpConnector is used for all WebClients.

    TelegramClient:

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8    @Autowired
    9    ReactorClientHttpConnector httpClient;
    10
    11    WebClient webClient;
    12
    13    RateLimiter rateLimiter;
    14
    15    @PostConstruct
    16    public void init() {
    17        webClient = WebClient.builder()
    18                .clientConnector(httpClient)
    19                .baseUrl(telegramUrl)
    20                .build();
    21
    22        rateLimiter = RateLimiter.of("telegram-rate-limiter",
    23                RateLimiterConfig.custom()
    24                        .limitRefreshPeriod(Duration.ofMinutes(1))
    25                        .limitForPeriod(20)
    26                        .build());
    27    }
    28
    29    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
    30        webClient.post().uri(String.format("/bot%s/sendMessage", token))
    31                .contentType(MediaType.APPLICATION_JSON)
    32                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
    33                        .with("text", message))
    34                .retrieve()
    35                .bodyToMono(Void.class)
    36                .transformDeferred(RateLimiterOperator.of(rateLimiter))
    37                .block();
    38    }
    39

    The RateLimiter is used to ensure the number of requests does not exceed 20 a minute as specified in the Telegram API.

    When the app is started, all requests are resolved normally as expected. But after some time has passed, all requests seem to get stuck. The amount of time needed for this to happen can vary from a few hours to a few days. It happens for all requests to different hosts and is easily noticeable when messages from the TelegramBot stops. Once the requests get stuck, they are stuck indefinitely and I have to restart the app to get it working again.

    There are no exceptions in the log that seem to have caused this. Since I maintain a queue for my telegram messages, I can see the point in time when the requests stop when the number of messages in the queue steadily increases and when errors happen in the other processes that are waiting for the requests to resolve.

    It does not seem like the requests are even sent out as the connect timeout and response timeout that I have set do not take effect.

    I had previously also tried setting idle time to 0 but that did not solve the problem

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8    @Autowired
    9    ReactorClientHttpConnector httpClient;
    10
    11    WebClient webClient;
    12
    13    RateLimiter rateLimiter;
    14
    15    @PostConstruct
    16    public void init() {
    17        webClient = WebClient.builder()
    18                .clientConnector(httpClient)
    19                .baseUrl(telegramUrl)
    20                .build();
    21
    22        rateLimiter = RateLimiter.of("telegram-rate-limiter",
    23                RateLimiterConfig.custom()
    24                        .limitRefreshPeriod(Duration.ofMinutes(1))
    25                        .limitForPeriod(20)
    26                        .build());
    27    }
    28
    29    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
    30        webClient.post().uri(String.format("/bot%s/sendMessage", token))
    31                .contentType(MediaType.APPLICATION_JSON)
    32                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
    33                        .with("text", message))
    34                .retrieve()
    35                .bodyToMono(Void.class)
    36                .transformDeferred(RateLimiterOperator.of(rateLimiter))
    37                .block();
    38    }
    39    @Bean
    40    public ReactorClientHttpConnector httpClient() {
    41        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
    42        HttpClient httpClient = HttpClient.newConnection()
    43                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    44                .responseTimeout(Duration.ofMillis(responseTimeout));
    45        return new ReactorClientHttpConnector(httpClient);
    46    }
    47

    Update:

    I enabled metrics and viewed it using micrometer when it got stuck. Interestingly, it shows that there is one connection for Telegram, but also shows no connections on idle, pending or active.

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8    @Autowired
    9    ReactorClientHttpConnector httpClient;
    10
    11    WebClient webClient;
    12
    13    RateLimiter rateLimiter;
    14
    15    @PostConstruct
    16    public void init() {
    17        webClient = WebClient.builder()
    18                .clientConnector(httpClient)
    19                .baseUrl(telegramUrl)
    20                .build();
    21
    22        rateLimiter = RateLimiter.of("telegram-rate-limiter",
    23                RateLimiterConfig.custom()
    24                        .limitRefreshPeriod(Duration.ofMinutes(1))
    25                        .limitForPeriod(20)
    26                        .build());
    27    }
    28
    29    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
    30        webClient.post().uri(String.format("/bot%s/sendMessage", token))
    31                .contentType(MediaType.APPLICATION_JSON)
    32                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
    33                        .with("text", message))
    34                .retrieve()
    35                .bodyToMono(Void.class)
    36                .transformDeferred(RateLimiterOperator.of(rateLimiter))
    37                .block();
    38    }
    39    @Bean
    40    public ReactorClientHttpConnector httpClient() {
    41        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
    42        HttpClient httpClient = HttpClient.newConnection()
    43                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    44                .responseTimeout(Duration.ofMillis(responseTimeout));
    45        return new ReactorClientHttpConnector(httpClient);
    46    }
    47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
    51

    Could the issue be this missing connection?

    Update 2:

    I thought this might be related to this other issue: Closing Reactor Netty connection on error status codes

    So I updated my HttpClient to this:

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8    @Autowired
    9    ReactorClientHttpConnector httpClient;
    10
    11    WebClient webClient;
    12
    13    RateLimiter rateLimiter;
    14
    15    @PostConstruct
    16    public void init() {
    17        webClient = WebClient.builder()
    18                .clientConnector(httpClient)
    19                .baseUrl(telegramUrl)
    20                .build();
    21
    22        rateLimiter = RateLimiter.of("telegram-rate-limiter",
    23                RateLimiterConfig.custom()
    24                        .limitRefreshPeriod(Duration.ofMinutes(1))
    25                        .limitForPeriod(20)
    26                        .build());
    27    }
    28
    29    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
    30        webClient.post().uri(String.format("/bot%s/sendMessage", token))
    31                .contentType(MediaType.APPLICATION_JSON)
    32                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
    33                        .with("text", message))
    34                .retrieve()
    35                .bodyToMono(Void.class)
    36                .transformDeferred(RateLimiterOperator.of(rateLimiter))
    37                .block();
    38    }
    39    @Bean
    40    public ReactorClientHttpConnector httpClient() {
    41        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
    42        HttpClient httpClient = HttpClient.newConnection()
    43                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    44                .responseTimeout(Duration.ofMillis(responseTimeout));
    45        return new ReactorClientHttpConnector(httpClient);
    46    }
    47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
    51    @Bean
    52    public ReactorClientHttpConnector httpClient() {
    53        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
    54                .doAfterResponseSuccess((r, c) -> c.dispose())
    55                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    56                .responseTimeout(Duration.ofMillis(responseTimeout));
    57        return new ReactorClientHttpConnector(httpClient);
    58    }
    59

    But all that seemed to do is accelerate the occurrence of the problem. Just like before, the active, pending and idle connections do not add up to the total connections. The total is always greater than the other 3 metrics added together.

    Update 3: I did a thread dump when the issue happened. There were a total of 74 threads, so I don't think the app is running out of threads.

    The dump for the Telegram thread:

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8    @Autowired
    9    ReactorClientHttpConnector httpClient;
    10
    11    WebClient webClient;
    12
    13    RateLimiter rateLimiter;
    14
    15    @PostConstruct
    16    public void init() {
    17        webClient = WebClient.builder()
    18                .clientConnector(httpClient)
    19                .baseUrl(telegramUrl)
    20                .build();
    21
    22        rateLimiter = RateLimiter.of("telegram-rate-limiter",
    23                RateLimiterConfig.custom()
    24                        .limitRefreshPeriod(Duration.ofMinutes(1))
    25                        .limitForPeriod(20)
    26                        .build());
    27    }
    28
    29    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
    30        webClient.post().uri(String.format("/bot%s/sendMessage", token))
    31                .contentType(MediaType.APPLICATION_JSON)
    32                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
    33                        .with("text", message))
    34                .retrieve()
    35                .bodyToMono(Void.class)
    36                .transformDeferred(RateLimiterOperator.of(rateLimiter))
    37                .block();
    38    }
    39    @Bean
    40    public ReactorClientHttpConnector httpClient() {
    41        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
    42        HttpClient httpClient = HttpClient.newConnection()
    43                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    44                .responseTimeout(Duration.ofMillis(responseTimeout));
    45        return new ReactorClientHttpConnector(httpClient);
    46    }
    47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
    51    @Bean
    52    public ReactorClientHttpConnector httpClient() {
    53        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
    54                .doAfterResponseSuccess((r, c) -> c.dispose())
    55                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    56                .responseTimeout(Duration.ofMillis(responseTimeout));
    57        return new ReactorClientHttpConnector(httpClient);
    58    }
    59"TelegramBot" #20 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=47154.24s tid=0x00007f6b28e73000 nid=0x1c waiting on condition  [0x00007f6aed6fb000]
    60   java.lang.Thread.State: WAITING (parking)
    61        at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
    62        - parking to wait for  <0x00000000fa865c80> (a java.util.concurrent.CountDownLatch$Sync)
    63        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
    64        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
    65        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1039)
    66        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1345)
    67        at java.util.concurrent.CountDownLatch.await(java.base@11.0.13/CountDownLatch.java:232)
    68        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
    69        at reactor.core.publisher.Mono.block(Mono.java:1707)
    70        at com.moon.arbitrage.cm.feign.TelegramClient.sendMessage(TelegramClient.java:59)
    71        at com.moon.arbitrage.cm.service.TelegramService.lambda$sendArbMessage$0(TelegramService.java:53)
    72        at com.moon.arbitrage.cm.service.TelegramService$$Lambda$1092/0x000000084070f840.run(Unknown Source)
    73        at com.moon.arbitrage.cm.service.TelegramService.task(TelegramService.java:82)
    74        at com.moon.arbitrage.cm.service.TelegramService$$Lambda$920/0x0000000840665040.run(Unknown Source)
    75        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
    76
    77   Locked ownable synchronizers:
    78        - None
    79

    The reactor worker threads:

    1    @Bean
    2    public ReactorClientHttpConnector httpClient() {
    3        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
    4                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    5                .responseTimeout(Duration.ofMillis(responseTimeout));
    6        return new ReactorClientHttpConnector(httpClient);
    7    }
    8    @Autowired
    9    ReactorClientHttpConnector httpClient;
    10
    11    WebClient webClient;
    12
    13    RateLimiter rateLimiter;
    14
    15    @PostConstruct
    16    public void init() {
    17        webClient = WebClient.builder()
    18                .clientConnector(httpClient)
    19                .baseUrl(telegramUrl)
    20                .build();
    21
    22        rateLimiter = RateLimiter.of("telegram-rate-limiter",
    23                RateLimiterConfig.custom()
    24                        .limitRefreshPeriod(Duration.ofMinutes(1))
    25                        .limitForPeriod(20)
    26                        .build());
    27    }
    28
    29    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
    30        webClient.post().uri(String.format("/bot%s/sendMessage", token))
    31                .contentType(MediaType.APPLICATION_JSON)
    32                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
    33                        .with("text", message))
    34                .retrieve()
    35                .bodyToMono(Void.class)
    36                .transformDeferred(RateLimiterOperator.of(rateLimiter))
    37                .block();
    38    }
    39    @Bean
    40    public ReactorClientHttpConnector httpClient() {
    41        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
    42        HttpClient httpClient = HttpClient.newConnection()
    43                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    44                .responseTimeout(Duration.ofMillis(responseTimeout));
    45        return new ReactorClientHttpConnector(httpClient);
    46    }
    47reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    48reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    49reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
    50reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
    51    @Bean
    52    public ReactorClientHttpConnector httpClient() {
    53        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
    54                .doAfterResponseSuccess((r, c) -> c.dispose())
    55                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
    56                .responseTimeout(Duration.ofMillis(responseTimeout));
    57        return new ReactorClientHttpConnector(httpClient);
    58    }
    59"TelegramBot" #20 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=47154.24s tid=0x00007f6b28e73000 nid=0x1c waiting on condition  [0x00007f6aed6fb000]
    60   java.lang.Thread.State: WAITING (parking)
    61        at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
    62        - parking to wait for  <0x00000000fa865c80> (a java.util.concurrent.CountDownLatch$Sync)
    63        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
    64        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
    65        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1039)
    66        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1345)
    67        at java.util.concurrent.CountDownLatch.await(java.base@11.0.13/CountDownLatch.java:232)
    68        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
    69        at reactor.core.publisher.Mono.block(Mono.java:1707)
    70        at com.moon.arbitrage.cm.feign.TelegramClient.sendMessage(TelegramClient.java:59)
    71        at com.moon.arbitrage.cm.service.TelegramService.lambda$sendArbMessage$0(TelegramService.java:53)
    72        at com.moon.arbitrage.cm.service.TelegramService$$Lambda$1092/0x000000084070f840.run(Unknown Source)
    73        at com.moon.arbitrage.cm.service.TelegramService.task(TelegramService.java:82)
    74        at com.moon.arbitrage.cm.service.TelegramService$$Lambda$920/0x0000000840665040.run(Unknown Source)
    75        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
    76
    77   Locked ownable synchronizers:
    78        - None
    79"reactor-http-epoll-1" #15 daemon prio=5 os_prio=0 cpu=810.44ms elapsed=47157.07s tid=0x00007f6b281c4000 nid=0x17 runnable  [0x00007f6b0c46c000]
    80   java.lang.Thread.State: RUNNABLE
    81        at io.netty.channel.epoll.Native.epollWait0(Native Method)
    82        at io.netty.channel.epoll.Native.epollWait(Native.java:177)
    83        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286)
    84        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
    85        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    86        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    87        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    88        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
    89
    90   Locked ownable synchronizers:
    91        - None
    92
    93"reactor-http-epoll-2" #16 daemon prio=5 os_prio=0 cpu=1312.16ms elapsed=47157.07s tid=0x00007f6b281c5000 nid=0x18 waiting on condition  [0x00007f6b0c369000]
    94   java.lang.Thread.State: WAITING (parking)
    95        at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
    96        - parking to wait for  <0x00000000fa865948> (a java.util.concurrent.CompletableFuture$Signaller)
    97        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
    98        at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
    99        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
    100        at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
    101        at java.util.concurrent.CompletableFuture.get(java.base@11.0.13/CompletableFuture.java:1998)
    102        at com.moon.arbitrage.cm.service.OrderService.reconcileOrder(OrderService.java:103)
    103        at com.moon.arbitrage.cm.service.BotService$BotTask.lambda$task$1(BotService.java:383)
    104        at com.moon.arbitrage.cm.service.BotService$BotTask$$Lambda$1161/0x00000008400af440.accept(Unknown Source)
    105        at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:171)
    106        at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
    107        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
    108        at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249)
    109        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
    110        at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
    111        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
    112        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
    113        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
    114        at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
    115        at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
    116        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
    117        at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
    118        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
    119        at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
    120        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
    121        at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
    122        at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
    123        at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)
    124        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:702)
    125        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
    126        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    127        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    128        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    129        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    130        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    131        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    132        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    133        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    134        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
    135        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
    136        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    137        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    138        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    139        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    140        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
    141        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
    142        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
    143        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
    144        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
    145        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    146        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    147        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    148        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    149        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    150        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    151        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    152        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    153        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
    154        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
    155        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    156        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    157        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    158        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    159        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
    160
    161   Locked ownable synchronizers:
    162        - None
    163
    164"reactor-http-epoll-3" #17 daemon prio=5 os_prio=0 cpu=171.84ms elapsed=47157.07s tid=0x00007f6b28beb000 nid=0x19 runnable  [0x00007f6b0c26a000]
    165   java.lang.Thread.State: RUNNABLE
    166        at io.netty.channel.epoll.Native.epollWait0(Native Method)
    167        at io.netty.channel.epoll.Native.epollWait(Native.java:177)
    168        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
    169        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
    170        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    171        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    172        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    173        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
    174
    175   Locked ownable synchronizers:
    176        - None
    177
    178"reactor-http-epoll-4" #18 daemon prio=5 os_prio=0 cpu=188.10ms elapsed=47157.07s tid=0x00007f6b28b7d800 nid=0x1a runnable  [0x00007f6b0c169000]
    179   java.lang.Thread.State: RUNNABLE
    180        at io.netty.channel.epoll.Native.epollWait0(Native Method)
    181        at io.netty.channel.epoll.Native.epollWait(Native.java:177)
    182        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
    183        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
    184        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    185        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    186        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    187        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
    188
    189   Locked ownable synchronizers:
    190        - None
    191

    Seems like one of them is blocked with another task (that isn't even from the Telegram service) but that shouldn't be an issue since the other three worker threads are runnable right?

    ANSWER

    Answered 2021-Dec-20 at 14:25

    I would propose to take a look in the RateLimiter direction. Maybe it does not work as expected, depending on the number of requests your application does over time. From the Javadoc for Ratelimiter: "It is important to note that the number of permits requested never affects the throttling of the request itself ... but it affects the throttling of the next request. I.e., if an expensive task arrives at an idle RateLimiter, it will be granted immediately, but it is the next request that will experience extra throttling, thus paying for the cost of the expensive task." Also helpful might be this discussion: github or github

    I could imaginge there is some throttling adding up or other effect in the RateLimiter, i would try to play around with it and make sure this thing really works the way you want. Alternatively, consider using Spring @Scheduled to read from your queue. You might want to spice it up using embedded JMS for further goodies (message persistence etc).

    Source https://stackoverflow.com/questions/70357582

    QUESTION

    How do I get mobile status for discord bot by directly modifying IDENTIFY packet?

    Asked 2022-Feb-09 at 15:05

    Apparently, discord bots can have mobile status as opposed to the desktop (online) status that one gets by default.

    bot having mobile status

    After a bit of digging I found out that such a status is achieved by modifying the IDENTIFY packet in discord.gateway.DiscordWebSocket.identify modifying the value of $browser to Discord Android or Discord iOS should theoretically get us the mobile status.

    After modifying code snippets I found online which does this, I end up with this :

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32

    Now all there is left to do is overwrite the discord.gateway.DiscordWebSocket.identify during runtime in the main file, something like this :

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32import discord
    33import os
    34from discord.ext import commands
    35import mobile_status
    36
    37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    38bot = commands.Bot(command_prefix="?")
    39
    40@bot.event
    41async def on_ready():
    42    print(f"Sucessfully logged in as {bot.user}")
    43
    44bot.run(os.getenv("DISCORD_TOKEN"))
    45

    And we do get the mobile status successfully
    successful mobile status for bot

    But here's the problem, I wanted to directly modify the file (which held the function) rather than monkey-patching it during runtime. So I cloned the dpy lib locally and edited the file on my machine, it ended up looking like this :

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32import discord
    33import os
    34from discord.ext import commands
    35import mobile_status
    36
    37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    38bot = commands.Bot(command_prefix="?")
    39
    40@bot.event
    41async def on_ready():
    42    print(f"Sucessfully logged in as {bot.user}")
    43
    44bot.run(os.getenv("DISCORD_TOKEN"))
    45    async def identify(self):
    46        """Sends the IDENTIFY packet."""
    47        payload = {
    48            'op': self.IDENTIFY,
    49            'd': {
    50                'token': self.token,
    51                'properties': {
    52                    '$os': sys.platform,
    53                    '$browser': 'Discord Android',
    54                    '$device': 'Discord Android',
    55                    '$referrer': '',
    56                    '$referring_domain': ''
    57                },
    58                'compress': True,
    59                'large_threshold': 250,
    60                'v': 3
    61            }
    62        }
    63     # ...
    64

    (edited both $browser and $device to Discord Android just to be safe)

    But this does not work and just gives me the regular desktop online icon.
    So the next thing I did is to inspect the identify function after it has been monkey-patched, so I could just look at the source code and see what went wrong earlier, but due to hard luck I got this error :

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32import discord
    33import os
    34from discord.ext import commands
    35import mobile_status
    36
    37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    38bot = commands.Bot(command_prefix="?")
    39
    40@bot.event
    41async def on_ready():
    42    print(f"Sucessfully logged in as {bot.user}")
    43
    44bot.run(os.getenv("DISCORD_TOKEN"))
    45    async def identify(self):
    46        """Sends the IDENTIFY packet."""
    47        payload = {
    48            'op': self.IDENTIFY,
    49            'd': {
    50                'token': self.token,
    51                'properties': {
    52                    '$os': sys.platform,
    53                    '$browser': 'Discord Android',
    54                    '$device': 'Discord Android',
    55                    '$referrer': '',
    56                    '$referring_domain': ''
    57                },
    58                'compress': True,
    59                'large_threshold': 250,
    60                'v': 3
    61            }
    62        }
    63     # ...
    64Traceback (most recent call last):
    65  File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
    66    print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    67  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
    68    lines, lnum = getsourcelines(object)
    69  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
    70    lines, lnum = findsource(object)
    71  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
    72    raise OSError('could not get source code')
    73OSError: could not get source code
    74

    Code :

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32import discord
    33import os
    34from discord.ext import commands
    35import mobile_status
    36
    37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    38bot = commands.Bot(command_prefix="?")
    39
    40@bot.event
    41async def on_ready():
    42    print(f"Sucessfully logged in as {bot.user}")
    43
    44bot.run(os.getenv("DISCORD_TOKEN"))
    45    async def identify(self):
    46        """Sends the IDENTIFY packet."""
    47        payload = {
    48            'op': self.IDENTIFY,
    49            'd': {
    50                'token': self.token,
    51                'properties': {
    52                    '$os': sys.platform,
    53                    '$browser': 'Discord Android',
    54                    '$device': 'Discord Android',
    55                    '$referrer': '',
    56                    '$referring_domain': ''
    57                },
    58                'compress': True,
    59                'large_threshold': 250,
    60                'v': 3
    61            }
    62        }
    63     # ...
    64Traceback (most recent call last):
    65  File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
    66    print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    67  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
    68    lines, lnum = getsourcelines(object)
    69  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
    70    lines, lnum = findsource(object)
    71  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
    72    raise OSError('could not get source code')
    73OSError: could not get source code
    74import discord
    75import os
    76from discord.ext import commands
    77import mobile_status
    78import inspect
    79
    80discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    81print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    82bot = commands.Bot(command_prefix="?")
    83
    84@bot.event
    85async def on_ready():
    86    print(f"Sucessfully logged in as {bot.user}")
    87
    88bot.run(os.getenv("DISCORD_TOKEN"))
    89

    Since this same behavior was exhibited for every patched function (aforementioned one and the loc["identify"]) I could no longer use inspect.getsource(...) and then relied upon dis.dis which lead to much more disappointing results

    The disassembled data looks exactly identical to the monkey-patched working version, so the directly modified version simply does not work despite function content being the exact same. (In regards to disassembled data)

    Notes: Doing Discord iOS directly does not work either, changing the $device to some other value but keeping $browser does not work, I have tried all combinations, none of them work.

    TL;DR: How to get mobile status for discord bot without monkey-patching it during runtime?

    ANSWER

    Answered 2022-Feb-07 at 23:03

    The following works by subclassing the relevant class, and duplicating code with the relevant changes. We also have to subclass the Client class, to overwrite the place where the gateway/websocket class is used. This results in a lot of duplicated code, however it does work, and requires neither dirty monkey-patching nor editing the library source code.

    However, it does come with many of the same problems as editing the library source code - mainly that as the library is updated, this code will become out of date (if you're using the archived and obsolete version of the library, you have bigger problems instead).

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32import discord
    33import os
    34from discord.ext import commands
    35import mobile_status
    36
    37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    38bot = commands.Bot(command_prefix="?")
    39
    40@bot.event
    41async def on_ready():
    42    print(f"Sucessfully logged in as {bot.user}")
    43
    44bot.run(os.getenv("DISCORD_TOKEN"))
    45    async def identify(self):
    46        """Sends the IDENTIFY packet."""
    47        payload = {
    48            'op': self.IDENTIFY,
    49            'd': {
    50                'token': self.token,
    51                'properties': {
    52                    '$os': sys.platform,
    53                    '$browser': 'Discord Android',
    54                    '$device': 'Discord Android',
    55                    '$referrer': '',
    56                    '$referring_domain': ''
    57                },
    58                'compress': True,
    59                'large_threshold': 250,
    60                'v': 3
    61            }
    62        }
    63     # ...
    64Traceback (most recent call last):
    65  File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
    66    print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    67  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
    68    lines, lnum = getsourcelines(object)
    69  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
    70    lines, lnum = findsource(object)
    71  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
    72    raise OSError('could not get source code')
    73OSError: could not get source code
    74import discord
    75import os
    76from discord.ext import commands
    77import mobile_status
    78import inspect
    79
    80discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    81print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    82bot = commands.Bot(command_prefix="?")
    83
    84@bot.event
    85async def on_ready():
    86    print(f"Sucessfully logged in as {bot.user}")
    87
    88bot.run(os.getenv("DISCORD_TOKEN"))
    89import asyncio
    90import sys
    91
    92import aiohttp
    93
    94import discord
    95from discord.gateway import DiscordWebSocket, _log
    96from discord.ext.commands import Bot
    97
    98
    99class MyGateway(DiscordWebSocket):
    100
    101    async def identify(self):
    102        payload = {
    103            'op': self.IDENTIFY,
    104            'd': {
    105                'token': self.token,
    106                'properties': {
    107                    '$os': sys.platform,
    108                    '$browser': 'Discord Android',
    109                    '$device': 'Discord Android',
    110                    '$referrer': '',
    111                    '$referring_domain': ''
    112                },
    113                'compress': True,
    114                'large_threshold': 250,
    115                'v': 3
    116            }
    117        }
    118
    119        if self.shard_id is not None and self.shard_count is not None:
    120            payload['d']['shard'] = [self.shard_id, self.shard_count]
    121
    122        state = self._connection
    123        if state._activity is not None or state._status is not None:
    124            payload['d']['presence'] = {
    125                'status': state._status,
    126                'game': state._activity,
    127                'since': 0,
    128                'afk': False
    129            }
    130
    131        if state._intents is not None:
    132            payload['d']['intents'] = state._intents.value
    133
    134        await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
    135        await self.send_as_json(payload)
    136        _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
    137
    138
    139class MyBot(Bot):
    140
    141    async def connect(self, *, reconnect: bool = True) -> None:
    142        """|coro|
    143
    144        Creates a websocket connection and lets the websocket listen
    145        to messages from Discord. This is a loop that runs the entire
    146        event system and miscellaneous aspects of the library. Control
    147        is not resumed until the WebSocket connection is terminated.
    148
    149        Parameters
    150        -----------
    151        reconnect: :class:`bool`
    152            If we should attempt reconnecting, either due to internet
    153            failure or a specific failure on Discord's part. Certain
    154            disconnects that lead to bad state will not be handled (such as
    155            invalid sharding payloads or bad tokens).
    156
    157        Raises
    158        -------
    159        :exc:`.GatewayNotFound`
    160            If the gateway to connect to Discord is not found. Usually if this
    161            is thrown then there is a Discord API outage.
    162        :exc:`.ConnectionClosed`
    163            The websocket connection has been terminated.
    164        """
    165
    166        backoff = discord.client.ExponentialBackoff()
    167        ws_params = {
    168            'initial': True,
    169            'shard_id': self.shard_id,
    170        }
    171        while not self.is_closed():
    172            try:
    173                coro = MyGateway.from_client(self, **ws_params)
    174                self.ws = await asyncio.wait_for(coro, timeout=60.0)
    175                ws_params['initial'] = False
    176                while True:
    177                    await self.ws.poll_event()
    178            except discord.client.ReconnectWebSocket as e:
    179                _log.info('Got a request to %s the websocket.', e.op)
    180                self.dispatch('disconnect')
    181                ws_params.update(sequence=self.ws.sequence, resume=e.resume, session=self.ws.session_id)
    182                continue
    183            except (OSError,
    184                    discord.HTTPException,
    185                    discord.GatewayNotFound,
    186                    discord.ConnectionClosed,
    187                    aiohttp.ClientError,
    188                    asyncio.TimeoutError) as exc:
    189
    190                self.dispatch('disconnect')
    191                if not reconnect:
    192                    await self.close()
    193                    if isinstance(exc, discord.ConnectionClosed) and exc.code == 1000:
    194                        # clean close, don't re-raise this
    195                        return
    196                    raise
    197
    198                if self.is_closed():
    199                    return
    200
    201                # If we get connection reset by peer then try to RESUME
    202                if isinstance(exc, OSError) and exc.errno in (54, 10054):
    203                    ws_params.update(sequence=self.ws.sequence, initial=False, resume=True, session=self.ws.session_id)
    204                    continue
    205
    206                # We should only get this when an unhandled close code happens,
    207                # such as a clean disconnect (1000) or a bad state (bad token, no sharding, etc)
    208                # sometimes, discord sends us 1000 for unknown reasons so we should reconnect
    209                # regardless and rely on is_closed instead
    210                if isinstance(exc, discord.ConnectionClosed):
    211                    if exc.code == 4014:
    212                        raise discord.PrivilegedIntentsRequired(exc.shard_id) from None
    213                    if exc.code != 1000:
    214                        await self.close()
    215                        raise
    216
    217                retry = backoff.delay()
    218                _log.exception("Attempting a reconnect in %.2fs", retry)
    219                await asyncio.sleep(retry)
    220                # Always try to RESUME the connection
    221                # If the connection is not RESUME-able then the gateway will invalidate the session.
    222                # This is apparently what the official Discord client does.
    223                ws_params.update(sequence=self.ws.sequence, resume=True, session=self.ws.session_id)
    224
    225
    226bot = MyBot(command_prefix="?")
    227
    228
    229@bot.event
    230async def on_ready():
    231    print(f"Sucessfully logged in as {bot.user}")
    232
    233bot.run("YOUR_BOT_TOKEN")
    234

    Personally, I think that the following approach, which does include some runtime monkey-patching (but no AST manipulation) is cleaner for this purpose:

    1def get_mobile():
    2    """
    3    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    4    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    5    from there they send the extended presence object.
    6    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    7    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    8    the mobile indicator is not triggered by the mobile client.
    9    The specific values for the $os, $browser, and $device fields are can change from time to time.
    10    """
    11    import ast
    12    import inspect
    13    import re
    14    import discord
    15
    16    def source(o):
    17        s = inspect.getsource(o).split("\n")
    18        indent = len(s[0]) - len(s[0].lstrip())
    19
    20        return "\n".join(i[indent:] for i in s)
    21
    22    source_ = source(discord.gateway.DiscordWebSocket.identify)
    23    patched = re.sub(
    24        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
    25        r"\1Discord Android\2",
    26        source_,
    27    )
    28
    29    loc = {}
    30    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    31    return loc["identify"]
    32import discord
    33import os
    34from discord.ext import commands
    35import mobile_status
    36
    37discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    38bot = commands.Bot(command_prefix="?")
    39
    40@bot.event
    41async def on_ready():
    42    print(f"Sucessfully logged in as {bot.user}")
    43
    44bot.run(os.getenv("DISCORD_TOKEN"))
    45    async def identify(self):
    46        """Sends the IDENTIFY packet."""
    47        payload = {
    48            'op': self.IDENTIFY,
    49            'd': {
    50                'token': self.token,
    51                'properties': {
    52                    '$os': sys.platform,
    53                    '$browser': 'Discord Android',
    54                    '$device': 'Discord Android',
    55                    '$referrer': '',
    56                    '$referring_domain': ''
    57                },
    58                'compress': True,
    59                'large_threshold': 250,
    60                'v': 3
    61            }
    62        }
    63     # ...
    64Traceback (most recent call last):
    65  File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
    66    print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    67  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
    68    lines, lnum = getsourcelines(object)
    69  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
    70    lines, lnum = findsource(object)
    71  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
    72    raise OSError('could not get source code')
    73OSError: could not get source code
    74import discord
    75import os
    76from discord.ext import commands
    77import mobile_status
    78import inspect
    79
    80discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
    81print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
    82bot = commands.Bot(command_prefix="?")
    83
    84@bot.event
    85async def on_ready():
    86    print(f"Sucessfully logged in as {bot.user}")
    87
    88bot.run(os.getenv("DISCORD_TOKEN"))
    89import asyncio
    90import sys
    91
    92import aiohttp
    93
    94import discord
    95from discord.gateway import DiscordWebSocket, _log
    96from discord.ext.commands import Bot
    97
    98
    99class MyGateway(DiscordWebSocket):
    100
    101    async def identify(self):
    102        payload = {
    103            'op': self.IDENTIFY,
    104            'd': {
    105                'token': self.token,
    106                'properties': {
    107                    '$os': sys.platform,
    108                    '$browser': 'Discord Android',
    109                    '$device': 'Discord Android',
    110                    '$referrer': '',
    111                    '$referring_domain': ''
    112                },
    113                'compress': True,
    114                'large_threshold': 250,
    115                'v': 3
    116            }
    117        }
    118
    119        if self.shard_id is not None and self.shard_count is not None:
    120            payload['d']['shard'] = [self.shard_id, self.shard_count]
    121
    122        state = self._connection
    123        if state._activity is not None or state._status is not None:
    124            payload['d']['presence'] = {
    125                'status': state._status,
    126                'game': state._activity,
    127                'since': 0,
    128                'afk': False
    129            }
    130
    131        if state._intents is not None:
    132            payload['d']['intents'] = state._intents.value
    133
    134        await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
    135        await self.send_as_json(payload)
    136        _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
    137
    138
    139class MyBot(Bot):
    140
    141    async def connect(self, *, reconnect: bool = True) -> None:
    142        """|coro|
    143
    144        Creates a websocket connection and lets the websocket listen
    145        to messages from Discord. This is a loop that runs the entire
    146        event system and miscellaneous aspects of the library. Control
    147        is not resumed until the WebSocket connection is terminated.
    148
    149        Parameters
    150        -----------
    151        reconnect: :class:`bool`
    152            If we should attempt reconnecting, either due to internet
    153            failure or a specific failure on Discord's part. Certain
    154            disconnects that lead to bad state will not be handled (such as
    155            invalid sharding payloads or bad tokens).
    156
    157        Raises
    158        -------
    159        :exc:`.GatewayNotFound`
    160            If the gateway to connect to Discord is not found. Usually if this
    161            is thrown then there is a Discord API outage.
    162        :exc:`.ConnectionClosed`
    163            The websocket connection has been terminated.
    164        """
    165
    166        backoff = discord.client.ExponentialBackoff()
    167        ws_params = {
    168            'initial': True,
    169            'shard_id': self.shard_id,
    170        }
    171        while not self.is_closed():
    172            try:
    173                coro = MyGateway.from_client(self, **ws_params)
    174                self.ws = await asyncio.wait_for(coro, timeout=60.0)
    175                ws_params['initial'] = False
    176                while True:
    177                    await self.ws.poll_event()
    178            except discord.client.ReconnectWebSocket as e:
    179                _log.info('Got a request to %s the websocket.', e.op)
    180                self.dispatch('disconnect')
    181                ws_params.update(sequence=self.ws.sequence, resume=e.resume, session=self.ws.session_id)
    182                continue
    183            except (OSError,
    184                    discord.HTTPException,
    185                    discord.GatewayNotFound,
    186                    discord.ConnectionClosed,
    187                    aiohttp.ClientError,
    188                    asyncio.TimeoutError) as exc:
    189
    190                self.dispatch('disconnect')
    191                if not reconnect:
    192                    await self.close()
    193                    if isinstance(exc, discord.ConnectionClosed) and exc.code == 1000:
    194                        # clean close, don't re-raise this
    195                        return
    196                    raise
    197
    198                if self.is_closed():
    199                    return
    200
    201                # If we get connection reset by peer then try to RESUME
    202                if isinstance(exc, OSError) and exc.errno in (54, 10054):
    203                    ws_params.update(sequence=self.ws.sequence, initial=False, resume=True, session=self.ws.session_id)
    204                    continue
    205
    206                # We should only get this when an unhandled close code happens,
    207                # such as a clean disconnect (1000) or a bad state (bad token, no sharding, etc)
    208                # sometimes, discord sends us 1000 for unknown reasons so we should reconnect
    209                # regardless and rely on is_closed instead
    210                if isinstance(exc, discord.ConnectionClosed):
    211                    if exc.code == 4014:
    212                        raise discord.PrivilegedIntentsRequired(exc.shard_id) from None
    213                    if exc.code != 1000:
    214                        await self.close()
    215                        raise
    216
    217                retry = backoff.delay()
    218                _log.exception("Attempting a reconnect in %.2fs", retry)
    219                await asyncio.sleep(retry)
    220                # Always try to RESUME the connection
    221                # If the connection is not RESUME-able then the gateway will invalidate the session.
    222                # This is apparently what the official Discord client does.
    223                ws_params.update(sequence=self.ws.sequence, resume=True, session=self.ws.session_id)
    224
    225
    226bot = MyBot(command_prefix="?")
    227
    228
    229@bot.event
    230async def on_ready():
    231    print(f"Sucessfully logged in as {bot.user}")
    232
    233bot.run("YOUR_BOT_TOKEN")
    234import sys
    235from discord.gateway import DiscordWebSocket, _log
    236from discord.ext.commands import Bot
    237
    238
    239async def identify(self):
    240    payload = {
    241        'op': self.IDENTIFY,
    242        'd': {
    243            'token': self.token,
    244            'properties': {
    245                '$os': sys.platform,
    246                '$browser': 'Discord Android',
    247                '$device': 'Discord Android',
    248                '$referrer': '',
    249                '$referring_domain': ''
    250            },
    251            'compress': True,
    252            'large_threshold': 250,
    253            'v': 3
    254        }
    255    }
    256
    257    if self.shard_id is not None and self.shard_count is not None:
    258        payload['d']['shard'] = [self.shard_id, self.shard_count]
    259
    260    state = self._connection
    261    if state._activity is not None or state._status is not None:
    262        payload['d']['presence'] = {
    263            'status': state._status,
    264            'game': state._activity,
    265            'since': 0,
    266            'afk': False
    267        }
    268
    269    if state._intents is not None:
    270        payload['d']['intents'] = state._intents.value
    271
    272    await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
    273    await self.send_as_json(payload)
    274    _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
    275
    276
    277DiscordWebSocket.identify = identify
    278bot = Bot(command_prefix="?")
    279
    280
    281@bot.event
    282async def on_ready():
    283    print(f"Sucessfully logged in as {bot.user}")
    284
    285bot.run("YOUR_DISCORD_TOKEN")
    286

    As to why editing the library source code did not work for you, I can only assume that you have edited the wrong copy of the file, as people have commented.

    Source https://stackoverflow.com/questions/70903401

    QUESTION

    Changing Category/Channels Permissions Returns Error "Missing Permissions" - Novus/Discord.py

    Asked 2022-Feb-08 at 01:39

    SOLVED - The categories/channels were set to private which is why they couldn't be changed even with the correct permissions. My solution was to tell the user which ones and to add the role with the permissions manually. I added a message that said they can just give the bot administrator permissions and it will do it for them. Hopefully, this helps anyone who runs into this in the future.

    I'm trying to change the permissions of multiple categories and channels using the following code:

    1role = get(guild.roles, name="Fun")
    2for channel in guild.channels:
    3    overwrites = channel.overwrites_for(role)
    4    overwrites.view_channel = False
    5    await channel.set_permissions(role, overwrite=overwrites)
    6

    I get the following error:

    1role = get(guild.roles, name="Fun")
    2for channel in guild.channels:
    3    overwrites = channel.overwrites_for(role)
    4    overwrites.view_channel = False
    5    await channel.set_permissions(role, overwrite=overwrites)
    6await channel.set_permissions(role, overwrite=overwrites)
    7discord.errors.Forbidden: 403 Forbidden (error code: 50013): Missing Permissions
    8

    It will work if I give the bot the administrator permission. However, it won't work if I give the bot every single other permission, the bot is at the top of the roles, and it has every category permission. What am I missing? I'm using Novus (fork of discord.py so it's pretty much the same thing).

    Edit: I've asked people in the discord.py, Novus, and discord developers discord servers and they can't help me either. With the exception of the discord developers server they just didn't answer my question.

    Edit 2: I was able to get it working when I manually added the bot's role to the category and enabled view_channel. This won't work, however, because that means the server owner has to manually do this for every category and channel which is really inconvenient.

    ANSWER

    Answered 2021-Oct-31 at 12:05

    This is simply how Discord works, if a channel is private and the bot does not have administrator permissions or role that has permissions to view the channel, it is not able to view it.

    You can create a new category that the bot can access like this:

    1role = get(guild.roles, name="Fun")
    2for channel in guild.channels:
    3    overwrites = channel.overwrites_for(role)
    4    overwrites.view_channel = False
    5    await channel.set_permissions(role, overwrite=overwrites)
    6await channel.set_permissions(role, overwrite=overwrites)
    7discord.errors.Forbidden: 403 Forbidden (error code: 50013): Missing Permissions
    8role = await guild.create_role(name="Fun")
    9category = await guild.create_category("category_name")
    10await category.set_permissions(role, read_messages=True, send_messages=True, connect=True, speak=True)
    11await category.set_permissions(ctx.guild.me, read_messages=True, send_messages=True, speak=True)
    12

    Source https://stackoverflow.com/questions/69694159

    QUESTION

    Error [ERR_REQUIRE_ESM]: require() of ES Module not supported

    Asked 2022-Feb-03 at 22:08

    I'm trying to make a Discord bot that just says if someone is online on the game.

    However I keep getting this message:

    [ERR_REQUIRE_ESM]: require() of ES Module from not supported. Instead change the require of index.js in... to a dynamic import() which is available in all CommonJS modules.

    This is my code:

    1    module.exports = {
    2        name: 'username',
    3        description: "this is the username command",
    4        async execute(message, args) {
    5
    6            const fetch = require('node-fetch');
    7
    8            if (args.length !== 1) {
    9                return message.channel.send("invalid username wtf")
    10            }
    11
    12            const ign = args[0]
    13
    14            if (ign.length > 16 || ign.length < 3) {
    15                return message.channel.send("invalid username wtf")
    16            }
    17
    18            const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
    19            const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
    20
    21            if (uuid.length !== 32) {
    22                return;
    23            }
    24
    25            if (onlineinfo.success) {
    26                if (onlineinfo.session.online) {
    27                    message.channel.send("they are online")
    28                }
    29                else {
    30                    message.channel.send("they are offline")
    31                }
    32            }
    33            else {
    34                message.channel.send("hypixel api bad wtf")
    35            }
    36        }
    37    }
    38

    This is my package.json file:

    1    module.exports = {
    2        name: 'username',
    3        description: "this is the username command",
    4        async execute(message, args) {
    5
    6            const fetch = require('node-fetch');
    7
    8            if (args.length !== 1) {
    9                return message.channel.send("invalid username wtf")
    10            }
    11
    12            const ign = args[0]
    13
    14            if (ign.length > 16 || ign.length < 3) {
    15                return message.channel.send("invalid username wtf")
    16            }
    17
    18            const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
    19            const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
    20
    21            if (uuid.length !== 32) {
    22                return;
    23            }
    24
    25            if (onlineinfo.success) {
    26                if (onlineinfo.session.online) {
    27                    message.channel.send("they are online")
    28                }
    29                else {
    30                    message.channel.send("they are offline")
    31                }
    32            }
    33            else {
    34                message.channel.send("hypixel api bad wtf")
    35            }
    36        }
    37    }
    38{
    39    "name": "discordbot",
    40    "version": "1.0.0",
    41    "main": "main.js",
    42    "scripts": {
    43        "test": "echo \"Error: no test specified\" && exit 1",
    44        "start": "node main.js"
    45    },
    46    "author": "",
    47    "license": "ISC",
    48    "description": "",
    49    "dependencies": {
    50        "discord.js": "^13.0.1",
    51        "node-fetch": "^3.0.0"
    52    }
    53}
    54

    ANSWER

    Answered 2021-Sep-07 at 06:38

    node-fetch v3 recently stopped support for the require way of importing it in favor of ES Modules. You'll need to use ESM imports now, like:

    1    module.exports = {
    2        name: 'username',
    3        description: "this is the username command",
    4        async execute(message, args) {
    5
    6            const fetch = require('node-fetch');
    7
    8            if (args.length !== 1) {
    9                return message.channel.send("invalid username wtf")
    10            }
    11
    12            const ign = args[0]
    13
    14            if (ign.length > 16 || ign.length < 3) {
    15                return message.channel.send("invalid username wtf")
    16            }
    17
    18            const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
    19            const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
    20
    21            if (uuid.length !== 32) {
    22                return;
    23            }
    24
    25            if (onlineinfo.success) {
    26                if (onlineinfo.session.online) {
    27                    message.channel.send("they are online")
    28                }
    29                else {
    30                    message.channel.send("they are offline")
    31                }
    32            }
    33            else {
    34                message.channel.send("hypixel api bad wtf")
    35            }
    36        }
    37    }
    38{
    39    "name": "discordbot",
    40    "version": "1.0.0",
    41    "main": "main.js",
    42    "scripts": {
    43        "test": "echo \"Error: no test specified\" && exit 1",
    44        "start": "node main.js"
    45    },
    46    "author": "",
    47    "license": "ISC",
    48    "description": "",
    49    "dependencies": {
    50        "discord.js": "^13.0.1",
    51        "node-fetch": "^3.0.0"
    52    }
    53}
    54import fetch from "node-fetch";
    55

    at the top of your file.

    Source https://stackoverflow.com/questions/69081410

    QUESTION

    How to check if a bot can DM a user

    Asked 2022-Jan-22 at 22:03

    If a user has the privacy setting "Allow direct messages from server members" turned off and a discord bot calls

    1await user.dm_channel.send("Hello there")
    2

    You'll get this error:

    1await user.dm_channel.send("Hello there")
    2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
    3

    I would like to check whether I can message a user without sending them a message. Trying to send a message and catching this error does not work for me, because I don't want a message to get sent in the event that the bot is allowed to message.

    I have tried this:

    1await user.dm_channel.send("Hello there")
    2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
    3print(user.dm_channel.permissions_for(bot).send_messages)
    4

    but it always returns True, even if the message is not permitted.

    I have also tried this:

    1await user.dm_channel.send("Hello there")
    2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
    3print(user.dm_channel.permissions_for(bot).send_messages)
    4channel = await user.create_dm()
    5if channel is None:
    6    ...
    7

    but unfortunately, it seems that "has permission to message user" and "has permission to create a dm channel" are considered different.

    EDIT

    To clarify the exact usage since there seems to be a bit of confusion, take this example. There is a server, and 3 users in question: Me, My Bot, and Steve. Steve has "Allow direct messages from server members" checked off.

    The bot has a command called !newgame which accepts a list of users and starts a game amongst them, which involves DMing some of the members of the game. Because of Steve's privacy settings, he cannot play the game (since the bot will need to message him). If I do

    1await user.dm_channel.send("Hello there")
    2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
    3print(user.dm_channel.permissions_for(bot).send_messages)
    4channel = await user.create_dm()
    5if channel is None:
    6    ...
    7!newgame @DJMcMayhem @Steve
    8

    I'd like to provide a response like:

    1await user.dm_channel.send("Hello there")
    2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
    3print(user.dm_channel.permissions_for(bot).send_messages)
    4channel = await user.create_dm()
    5if channel is None:
    6    ...
    7!newgame @DJMcMayhem @Steve
    8> I can't start a game with that list of users because @Steve has the wrong privacy settings.
    9

    But as far as I know right now, the only way to find out if Steve can play is by first attempting to message every user, which I'd like to avoid.

    ANSWER

    Answered 2022-Jan-22 at 22:03
    Explanation

    You can generate a Bad Request to the dm_channel. This can be accomplished by setting content to None, for example.

    If it returns with 400 Bad Request, you can DM them. If it returns with 403 Forbidden, you can't.

    Code
    1await user.dm_channel.send("Hello there")
    2discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
    3print(user.dm_channel.permissions_for(bot).send_messages)
    4channel = await user.create_dm()
    5if channel is None:
    6    ...
    7!newgame @DJMcMayhem @Steve
    8> I can't start a game with that list of users because @Steve has the wrong privacy settings.
    9async def can_dm_user(user: discord.User) -> bool:
    10    ch = user.dm_channel
    11    if ch is None:
    12        ch = await user.create_dm()
    13
    14    try:
    15        await ch.send()
    16    except discord.Forbidden:
    17        return False
    18    except discord.HTTPException:
    19        return True
    20
    21

    Source https://stackoverflow.com/questions/70660854

    QUESTION

    How can I send Dynamic website content to scrapy with the html content generated by selenium browser?

    Asked 2022-Jan-20 at 15:35

    I am working on certain stock-related projects where I have had a task to scrape all data on a daily basis for the last 5 years. i.e from 2016 to date. I particularly thought of using selenium because I can use crawler and bot to scrape the data based on the date. So I used the use of button click with selenium and now I want the same data that is displayed by the selenium browser to be fed by scrappy. This is the website I am working on right now. I have written the following code inside scrappy spider.

    1class FloorSheetSpider(scrapy.Spider):
    2    name = "nepse"
    3
    4    def start_requests(self):
    5
    6        driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    7        
    8     
    9        floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
    10
    11        for date in floorsheet_dates:
    12            driver.get(
    13                "https://merolagani.com/Floorsheet.aspx")
    14
    15            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    16                                ).send_keys(date)
    17            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    18            total_length = driver.find_element(By.XPATH,
    19                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    20            z = int((total_length.split()[-1]).replace(']', ''))    
    21            for data in range(z, z + 1):
    22                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    23                self.url = driver.page_source
    24                yield Request(url=self.url, callback=self.parse)
    25
    26               
    27    def parse(self, response, **kwargs):
    28        for value in response.xpath('//tbody/tr'):
    29            print(value.css('td::text').extract()[1])
    30            print("ok"*200)
    31

    Update: Error after answer is

    1class FloorSheetSpider(scrapy.Spider):
    2    name = "nepse"
    3
    4    def start_requests(self):
    5
    6        driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    7        
    8     
    9        floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
    10
    11        for date in floorsheet_dates:
    12            driver.get(
    13                "https://merolagani.com/Floorsheet.aspx")
    14
    15            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    16                                ).send_keys(date)
    17            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    18            total_length = driver.find_element(By.XPATH,
    19                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    20            z = int((total_length.split()[-1]).replace(']', ''))    
    21            for data in range(z, z + 1):
    22                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    23                self.url = driver.page_source
    24                yield Request(url=self.url, callback=self.parse)
    25
    26               
    27    def parse(self, response, **kwargs):
    28        for value in response.xpath('//tbody/tr'):
    29            print(value.css('td::text').extract()[1])
    30            print("ok"*200)
    312022-01-14 14:11:36 [twisted] CRITICAL: 
    32Traceback (most recent call last):
    33  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    34    result = current_context.run(gen.send, result)
    35  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
    36    start_requests = iter(self.spider.start_requests())
    37TypeError: 'NoneType' object is not iterable
    38

    I want to send current web html content to scrapy feeder but I am getting unusal error for past 2 days any help or suggestions will be very much appreciated.

    ANSWER

    Answered 2022-Jan-14 at 09:30

    The 2 solutions are not very different. Solution #2 fits better to your question, but choose whatever you prefer.

    Solution 1 - create a response with the html's body from the driver and scraping it right away (you can also pass it as an argument to a function):

    1class FloorSheetSpider(scrapy.Spider):
    2    name = "nepse"
    3
    4    def start_requests(self):
    5
    6        driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    7        
    8     
    9        floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
    10
    11        for date in floorsheet_dates:
    12            driver.get(
    13                "https://merolagani.com/Floorsheet.aspx")
    14
    15            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    16                                ).send_keys(date)
    17            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    18            total_length = driver.find_element(By.XPATH,
    19                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    20            z = int((total_length.split()[-1]).replace(']', ''))    
    21            for data in range(z, z + 1):
    22                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    23                self.url = driver.page_source
    24                yield Request(url=self.url, callback=self.parse)
    25
    26               
    27    def parse(self, response, **kwargs):
    28        for value in response.xpath('//tbody/tr'):
    29            print(value.css('td::text').extract()[1])
    30            print("ok"*200)
    312022-01-14 14:11:36 [twisted] CRITICAL: 
    32Traceback (most recent call last):
    33  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    34    result = current_context.run(gen.send, result)
    35  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
    36    start_requests = iter(self.spider.start_requests())
    37TypeError: 'NoneType' object is not iterable
    38import scrapy
    39from selenium import webdriver
    40from selenium.webdriver.common.by import By
    41from scrapy.http import HtmlResponse
    42
    43
    44class FloorSheetSpider(scrapy.Spider):
    45    name = "nepse"
    46
    47    def start_requests(self):
    48
    49        # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    50        driver = webdriver.Chrome()
    51
    52        floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
    53
    54        for date in floorsheet_dates:
    55            driver.get(
    56                "https://merolagani.com/Floorsheet.aspx")
    57
    58            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    59                                ).send_keys(date)
    60            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    61            total_length = driver.find_element(By.XPATH,
    62                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    63            z = int((total_length.split()[-1]).replace(']', ''))
    64            for data in range(1, z + 1):
    65                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    66                self.body = driver.page_source
    67
    68                response = HtmlResponse(url=driver.current_url, body=self.body, encoding='utf-8')
    69                for value in response.xpath('//tbody/tr'):
    70                    print(value.css('td::text').extract()[1])
    71                    print("ok"*200)
    72
    73        # return an empty requests list
    74        return []
    75

    Solution 2 - with super simple downloader middleware:

    (You might have a delay here in parse method so be patient).

    1class FloorSheetSpider(scrapy.Spider):
    2    name = "nepse"
    3
    4    def start_requests(self):
    5
    6        driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    7        
    8     
    9        floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
    10
    11        for date in floorsheet_dates:
    12            driver.get(
    13                "https://merolagani.com/Floorsheet.aspx")
    14
    15            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    16                                ).send_keys(date)
    17            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    18            total_length = driver.find_element(By.XPATH,
    19                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    20            z = int((total_length.split()[-1]).replace(']', ''))    
    21            for data in range(z, z + 1):
    22                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    23                self.url = driver.page_source
    24                yield Request(url=self.url, callback=self.parse)
    25
    26               
    27    def parse(self, response, **kwargs):
    28        for value in response.xpath('//tbody/tr'):
    29            print(value.css('td::text').extract()[1])
    30            print("ok"*200)
    312022-01-14 14:11:36 [twisted] CRITICAL: 
    32Traceback (most recent call last):
    33  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    34    result = current_context.run(gen.send, result)
    35  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
    36    start_requests = iter(self.spider.start_requests())
    37TypeError: 'NoneType' object is not iterable
    38import scrapy
    39from selenium import webdriver
    40from selenium.webdriver.common.by import By
    41from scrapy.http import HtmlResponse
    42
    43
    44class FloorSheetSpider(scrapy.Spider):
    45    name = "nepse"
    46
    47    def start_requests(self):
    48
    49        # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    50        driver = webdriver.Chrome()
    51
    52        floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
    53
    54        for date in floorsheet_dates:
    55            driver.get(
    56                "https://merolagani.com/Floorsheet.aspx")
    57
    58            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    59                                ).send_keys(date)
    60            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    61            total_length = driver.find_element(By.XPATH,
    62                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    63            z = int((total_length.split()[-1]).replace(']', ''))
    64            for data in range(1, z + 1):
    65                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    66                self.body = driver.page_source
    67
    68                response = HtmlResponse(url=driver.current_url, body=self.body, encoding='utf-8')
    69                for value in response.xpath('//tbody/tr'):
    70                    print(value.css('td::text').extract()[1])
    71                    print("ok"*200)
    72
    73        # return an empty requests list
    74        return []
    75import scrapy
    76from scrapy import Request
    77from scrapy.http import HtmlResponse
    78from selenium import webdriver
    79from selenium.webdriver.common.by import By
    80
    81
    82class SeleniumMiddleware(object):
    83    def process_request(self, request, spider):
    84        url = spider.driver.current_url
    85        body = spider.driver.page_source
    86        return HtmlResponse(url=url, body=body, encoding='utf-8', request=request)
    87
    88
    89class FloorSheetSpider(scrapy.Spider):
    90    name = "nepse"
    91
    92    custom_settings = {
    93        'DOWNLOADER_MIDDLEWARES': {
    94            'tempbuffer.spiders.yetanotherspider.SeleniumMiddleware': 543,
    95            # 'projects_name.path.to.your.pipeline': 543
    96        }
    97    }
    98    driver = webdriver.Chrome()
    99
    100    def start_requests(self):
    101
    102        # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
    103
    104
    105        floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
    106
    107        for date in floorsheet_dates:
    108            self.driver.get(
    109                "https://merolagani.com/Floorsheet.aspx")
    110
    111            self.driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
    112                                ).send_keys(date)
    113            self.driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
    114            total_length = self.driver.find_element(By.XPATH,
    115                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
    116            z = int((total_length.split()[-1]).replace(']', ''))
    117            for data in range(1, z + 1):
    118                self.driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
    119                self.body = self.driver.page_source
    120                self.url = self.driver.current_url
    121
    122                yield Request(url=self.url, callback=self.parse, dont_filter=True)
    123
    124    def parse(self, response, **kwargs):
    125        print('test ok')
    126        for value in response.xpath('//tbody/tr'):
    127            print(value.css('td::text').extract()[1])
    128            print("ok"*200)
    129

    Notice that I've used chrome so change it back to firefox like in your original code.

    Source https://stackoverflow.com/questions/70651053

    QUESTION

    Python/Selenium web scrap how to find hidden src value from a links?

    Asked 2022-Jan-16 at 02:28

    Scrapping links should be a simple feat, usually just grabbing the src value of the a tag.

    I recently came across this website (https://sunteccity.com.sg/promotions) where the href value of a tags of each item cannot be found, but the redirection still works. I'm trying to figure out a way to grab the items and their corresponding links. My typical python selenium code looks something as such

    1all_items = bot.find_elements_by_class_name('thumb-img')
    2for promo in all_items:
    3    a = promo.find_elements_by_tag_name("a")
    4    print("a[0]: ", a[0].get_attribute("href"))
    5

    However, I can't seem to retrieve any href, onclick attributes, and I'm wondering if this is even possible. I noticed that I couldn't do a right-click, open link in new tab as well.

    Are there any ways around getting the links of all these items?

    Edit: Are there any ways to retrieve all the links of the items on the pages?

    i.e.

    1all_items = bot.find_elements_by_class_name('thumb-img')
    2for promo in all_items:
    3    a = promo.find_elements_by_tag_name("a")
    4    print("a[0]: ", a[0].get_attribute("href"))
    5https://sunteccity.com.sg/promotions/724
    6https://sunteccity.com.sg/promotions/731
    7https://sunteccity.com.sg/promotions/751
    8https://sunteccity.com.sg/promotions/752
    9https://sunteccity.com.sg/promotions/754
    10https://sunteccity.com.sg/promotions/280
    11...
    12

    Edit: Adding an image of one such anchor tag for better clarity: enter image description here

    ANSWER

    Answered 2022-Jan-15 at 19:47

    You are using a wrong locator. It brings you a lot of irrelevant elements.
    Instead of find_elements_by_class_name('thumb-img') please try find_elements_by_css_selector('.collections-page .thumb-img') so your code will be

    1all_items = bot.find_elements_by_class_name('thumb-img')
    2for promo in all_items:
    3    a = promo.find_elements_by_tag_name("a")
    4    print("a[0]: ", a[0].get_attribute("href"))
    5https://sunteccity.com.sg/promotions/724
    6https://sunteccity.com.sg/promotions/731
    7https://sunteccity.com.sg/promotions/751
    8https://sunteccity.com.sg/promotions/752
    9https://sunteccity.com.sg/promotions/754
    10https://sunteccity.com.sg/promotions/280
    11...
    12all_items = bot.find_elements_by_css_selector('.collections-page .thumb-img')
    13for promo in all_items:
    14    a = promo.find_elements_by_tag_name("a")
    15    print("a[0]: ", a[0].get_attribute("href"))
    16

    You can also get the desired links directly by .collections-page .thumb-img a locator so that your code could be:

    1all_items = bot.find_elements_by_class_name('thumb-img')
    2for promo in all_items:
    3    a = promo.find_elements_by_tag_name("a")
    4    print("a[0]: ", a[0].get_attribute("href"))
    5https://sunteccity.com.sg/promotions/724
    6https://sunteccity.com.sg/promotions/731
    7https://sunteccity.com.sg/promotions/751
    8https://sunteccity.com.sg/promotions/752
    9https://sunteccity.com.sg/promotions/754
    10https://sunteccity.com.sg/promotions/280
    11...
    12all_items = bot.find_elements_by_css_selector('.collections-page .thumb-img')
    13for promo in all_items:
    14    a = promo.find_elements_by_tag_name("a")
    15    print("a[0]: ", a[0].get_attribute("href"))
    16links = bot.find_elements_by_css_selector('.collections-page .thumb-img a')
    17for link in links:
    18    print(link.get_attribute("href"))
    19

    Source https://stackoverflow.com/questions/70721360

    Community Discussions contain sources that include Stack Exchange Network

    Tutorials and Learning Resources in Bot

    Share this Page

    share link

    Get latest updates on Bot