python-aria-mirror-bot | A telegram bot for all your mirror needs | Bot library

by lzzy12 Python Version: v3.1 License: GPL-3.0

Download this library from

kandi X-RAY | python-aria-mirror-bot Summary

python-aria-mirror-bot is a Python library typically used in Automation, Bot applications. python-aria-mirror-bot has no bugs, it has no vulnerabilities, it has build file available, it has a Strong Copyleft License and it has low support. You can download it from GitHub.

This is a telegram bot writen in python for mirroring files on the internet to our beloved Google Drive.

Support

Quality

Security

License

Reuse

kandi-support Support

python-aria-mirror-bot has a low active ecosystem.
It has 499 star(s) with 1126 fork(s). There are 40 watchers for this library.
It had no major release in the last 12 months.
There are 20 open issues and 138 have been closed. On average issues are closed in 87 days. There are 3 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of python-aria-mirror-bot is v3.1

This Library - Support

Best in #Bot

Average in #Bot

This Library - Support

Best in #Bot

Average in #Bot

quality kandi Quality

python-aria-mirror-bot has 0 bugs and 0 code smells.

This Library - Quality

Best in #Bot

Average in #Bot

This Library - Quality

Best in #Bot

Average in #Bot

security Security

python-aria-mirror-bot has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
python-aria-mirror-bot code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

This Library - Security

Best in #Bot

Average in #Bot

This Library - Security

Best in #Bot

Average in #Bot

license License

python-aria-mirror-bot is licensed under the GPL-3.0 License. This license is Strong Copyleft.
Strong Copyleft licenses enforce sharing, and you can use them when creating open source projects.

This Library - License

Best in #Bot

Average in #Bot

This Library - License

Best in #Bot

Average in #Bot

build Reuse

python-aria-mirror-bot releases are available to install and integrate.
Build file is available. You can build the component from source.
Installation instructions, examples and code snippets are available.
python-aria-mirror-bot saves you 1038 person hours of effort in developing the same functionality from scratch.
It has 2355 lines of code, 219 functions and 40 files.
It has high code complexity. Code complexity directly impacts maintainability of the code.

This Library - Reuse

Best in #Bot

Average in #Bot

This Library - Reuse

Best in #Bot

Average in #Bot

Top functions reviewed by kandi - BETA

kandi has reviewed python-aria-mirror-bot and discovered the below as its top functions. This is intended to give you an instant insight into python-aria-mirror-bot implemented functionality, and help decide if they suit your requirements.

Return a service account factory .
Uploads a file .
Called when the download has completed .
Mirror a message .
Cancel a mirror command .
Extract meta data from youtube .
Start the bot .
Help message for bot .
Watch the given link .
Create keys from projects .

python-aria-mirror-bot Key Features

Mirroring direct download links to google drive

Download progress

Upload progress

Download/upload speeds and ETAs

Docker support

Uploading To Team Drives.

Index Link support

Service account support

Mirror all youtube-dl supported links

Mirror telegram files

Stable Mega.nz support

python-aria-mirror-bot Examples and Code Snippets

See all related Code Snippets

Installing requirements

Copy

Download

git clone https://github.com/lzzy12/python-aria-mirror-bot mirror-bot/
cd mirror-bot

Setting up config file

Copy

Download

cp config_sample.env config.env

Getting Google OAuth API credential file

Copy

Download

pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
python3 generate_drive_token.py

Deploying

Copy

Download

sudo dockerd

Step 1. Generate service accounts

Copy

Download

Note: 1 service account can copy around 750gb a day, 1 project can make 100 service accounts so that's 75tb a day, for most users this should easily suffice.

Add all the service accounts to the Team Drive

Copy

Download

python3 add_to_team_drive.py -d SharedTeamDriveSrcID

Youtube-dl authentication using .netrc file

Copy

Download

machine host login username password my_youtube_password

See all related Code Snippets

Community Discussions

Trending Discussions on Bot

How can you create a pop-up window in Discord that accepts an input from the user?
Python Selenium AWS Lambda Change WebGL Vendor/Renderer For Undetectable Headless Scraper
Is there a way to access the children of a CategoryChannel before it is deleted? [Discord.js]
Spring Boot WebClient stops sending requests
How do I get mobile status for discord bot by directly modifying IDENTIFY packet?
Changing Category/Channels Permissions Returns Error "Missing Permissions" - Novus/Discord.py
Error [ERR_REQUIRE_ESM]: require() of ES Module not supported
How to check if a bot can DM a user
How can I send Dynamic website content to scrapy with the html content generated by selenium browser?
Python/Selenium web scrap how to find hidden src value from a links?

Trending Discussions on Bot

QUESTION

How can you create a pop-up window in Discord that accepts an input from the user?

Asked 2022-Mar-30 at 07:14

It's my first time seeing this feature from a Discord bot. I tried looking everywhere but it seems that I have failed. There's this feature from Captcha.bot Discord bot where you can accept input from a pop-up window inside Discord.

There's a button in an embedded message made by Captcha.bot where you will have to answer a Captcha test. After pressing the button, it creates a pop-up window like this.

After placing the right answer on the captcha bot, here's the aftermath of the experience.

All I want to learn is how to summon that pop-up window using Discord.js if it's even possible or at least learn how they did it.

ANSWER

Answered 2022-Mar-30 at 07:12

Those are called modals, and they will be available in the next discord.js version, v14. There is already a pull request for this.

In the meantime, you can use an npm package like discord-modals or discordjs-modal.

You can find a working example with the discord-modals package below. Don't forget to install it first using npm i discord-modals.

const {
  Client,
  Intents,
  MessageActionRow,
  MessageButton,
} = require('discord.js');
const discordModals = require('discord-modals');
const { Modal, TextInputComponent, showModal } = discordModals;

const TOKEN = 'YOUR TOKEN HERE';
const client = new Client({
  intents: [Intents.FLAGS.GUILDS, Intents.FLAGS.GUILD_MESSAGES],
});
discordModals(client);

client.on('messageCreate', (message) => {
  if (message.author.bot) return;

  let button = new MessageActionRow();
  button.addComponents(
    new MessageButton()
      .setCustomId('verification-button')
      .setStyle('PRIMARY')
      .setLabel('Open modal dialog'),
  );
  message.reply({
    components: [button],
  });
});

client.on('interactionCreate', async (interaction) => {
  if (interaction.isButton()) {
    if (interaction.customId === 'verification-button') {
      const modal = new Modal() // We create a Modal
        .setCustomId('verification-modal')
        .setTitle('Verify yourself')
        .addComponents([
          new TextInputComponent()
            .setCustomId('verification-input')
            .setLabel('Answer')
            .setStyle('SHORT')
            .setMinLength(4)
            .setMaxLength(12)
            .setPlaceholder('ABCDEF')
            .setRequired(true),
        ]);

      showModal(modal, {
        client,
        interaction,
      });
    }
  }
});

client.on('modalSubmit', async (modal) => {
  if (modal.customId === 'verification-modal') {
    const response = modal.getTextInputValue('verification-input');
    modal.reply(`Yay, your answer is submitted: "${response}"`);
  }
});

client.once('ready', () => {
  console.log('Bot v13 is connected...');
});

client.login(TOKEN);

Source https://stackoverflow.com/questions/71672321

QUESTION

Python Selenium AWS Lambda Change WebGL Vendor/Renderer For Undetectable Headless Scraper

Asked 2022-Mar-21 at 20:19

Concept:

Using AWS Lambda functions with Python and Selenium, I want to create a undetectable headless chrome scraper by passing a headless chrome test. I check the undetectability of my headless scraper by opening up the test and taking a screenshot. I ran this test on a Local IDE and on a Lambda server.

Implementation:

I will be using a python library called selenium-stealth and will follow their basic configuration:

stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        webgl_vendor="Intel Inc.",
        renderer="Intel Iris OpenGL Engine",
        fix_hairline=True,
        )

I implemented this configuration on a Local IDE as well as an AWS Lambda Server to compare the results.

Local IDE:

Found below are the test results running on a local IDE:

Lambda Server:

When I run this on a Lambda server, both the WebGL Vendor and Renderer are blank. as shown below:

I even tried to manually change the WebGL Vendor/Renderer using the following JavaScript command:

driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})

Then I thought maybe that it could be something wrong with the parameter number. I configured the command execution without the if statement, but the same thing happened: It worked on my Local IDE but had no effect on an AWS Lambda Server.

Simply Put:

Is it possible to add Vendor/Renderer on AWS Lambda? In my efforts, it seems that there is no possible way. I made sure to submit this issue on the selenium-stealth GitHub Repository.

ANSWER

Answered 2021-Dec-18 at 02:01

WebGL

WebGL is a cross-platform, open web standard for a low-level 3D graphics API based on OpenGL ES, exposed to ECMAScript via the HTML5 Canvas element. WebGL at it's core is a Shader-based API using GLSL, with constructs that are semantically similar to those of the underlying OpenGL ES API. It follows the OpenGL ES specification, with some exceptions for the out of memory-managed languages such as JavaScript. WebGL 1.0 exposes the OpenGL ES 2.0 feature set; WebGL 2.0 exposes the OpenGL ES 3.0 API.

Now, with the availability of Selenium Stealth building of Undetectable Scraper using Selenium driven ChromeDriver initiated google-chrome Browsing Context have become much more easier.

selenium-stealth

selenium-stealth is a python package selenium-stealth to prevent detection. This programme tries to make python selenium more stealthy. However, as of now selenium-stealth only support Selenium Chrome.

Code Block:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium_stealth import stealth

options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)

# Selenium Stealth settings
stealth(driver,
      languages=["en-US", "en"],
      vendor="Google Inc.",
      platform="Win32",
      webgl_vendor="Intel Inc.",
      renderer="Intel Iris OpenGL Engine",
      fix_hairline=True,
  )

driver.get("https://bot.sannysoft.com/")

Browser Screenshot:

bot_sannysoft

You can find a detailed relevant discussion in Can a website detect when you are using Selenium with chromedriver?

Changing WebGL Vendor/Renderer in AWS Lambda

AWS Lambda enables us to deliver compressed WebGL websites to end users. When requested webpage objects are compressed, the transfer size is reduced, leading to faster downloads, lower cloud storage fees, and lower data transfer fees. Improved load times also directly influence the viewer experience and retention, which helps in improving website conversion and discoverability. Using WebGL, websites are more immersive while still being accessible via a browser URL. Through this technique AWS Lambda to automatically compress the objects uploaded to S3.

Background on compression and WebGL

HTTP compression is a capability that can be built into web servers and web clients to improve transfer speed and bandwidth utilization. This capability is negotiated between the server and the client using an HTTP header which may indicate that a resource being transferred, cached, or otherwise referenced is compressed. AWS Lambda on the server-side supports Content-Encoding header.

On the client-side, most browsers today support brotli and gzip compression through HTTP headers (Accept-Encoding: deflate, br, gzip) and can handle server response headers. This means browsers will automatically download and decompress content from a web server at the client-side, before rendering webpages to the viewer.

Conclusion

Due to this constraint you may not be able to change the WebGL Vendor/Renderer in AWS Lambda, else it may directly affect the process of rendering webpages to the viewers and can stand out to be a bottleneck in UX.

tl; dr

You can find a couple of relevant detailed discussion in:

Can a website detect when you are using Selenium with chromedriver?

Source https://stackoverflow.com/questions/70265306

QUESTION

Is there a way to access the children of a CategoryChannel before it is deleted? [Discord.js]

Asked 2022-Feb-19 at 14:09

I handle a channelDelete event in my discord bot. My original intent was to do the following:

Listen for when a channel is deleted
Check to see if its type equals 'GUILD_CATEGORY'
Delete all the channels under that category

I can typically access channels under a CategoryChannel through its property called children anywhere else except during this event...

module.exports = {
    name: 'channelDelete',
    once: false,
    async execute(channel) {
        if(channel.type == 'GUILD_CATEGORY')
            for(let [child_id, child] of channel.children)
                if (child.deletable) child.delete();
    },
}

I can confirm the code is being executed, but the problem is that the incoming channel object is in a state where it is already deleted, and I cannot get the children:

During debugging, I noticed the channel object has the property deleted: true
The children property is empty, even though I know there were channels in that channel category prior to deletion

Is there a way for my bot to collect and handle the children of a CategoryChannel prior to its deletion?

ANSWER

Answered 2022-Feb-19 at 14:09

Why?

Unfortunately, this is how CategoryChannels work in discord.js...
When the category is deleted, discord.js sends a request to the API to delete the channel. Only then, Discord sends you the event after the category is deleted.
What happens next is that the children are not located in the category anymore! So you will not be able to get the children inside the CategoryChannel object.

This is the code for the children property

get children() {
    return this.guild.channels.cache.filter(c => c.parentId === this.id);
}

It filters the channels which are in the guild, but those children channels are not located in the category anymore. Which is the reason the property is empty.

The (Only?) Solution

This will require caching the channels themselves. As there aren't really any other solutions.
You may cache the channels in different ways.
Just remember... there are instances and references in Javascript, and failure to acknowledge may lead to weird behaviors. The following code only works for small bots without sharding, just so you know.

const cachedChannels = new Map()

client.on('ready', () => {
  // Looping thru guilds...
  client.guilds.forEach((guild) => {
    // Filtering the channels to category channels only and looping thru them...
    guild.channels.filter((channel) => channel.type === "GUILD_CATEGORY").forEach((category) => {
      // Note on references and instances: Numbers, Strings and Booleans do not have instances like Object, Arrays, etc. do
      // We are using an array here to abuse the references
      cachedChannels.set(category.id, [])

      // Looping thru the children channels...
      category.children.forEach((children) => {
        // This is the array we've created earlier
        const storedChannels = cachedChannels.get(category.id)

        // Pushing the ID so we can fetch them later
        storedChannels.push(children.id)
      })
    })
  })
})

client.on('channelDelete', (channel) => {
  if (channel.type === "GUILD_CATEGORY") {
    // Looping thru our cached channel IDs defined earlier
    cachedChannels.get(channel.id).forEach((id) => {
      const child = channel.guild.channels.get(id)
      child.delete()
    })

    // Delete the cached IDs to save memory
    cachedChannels.delete(channel.id)
  }
})

Source https://stackoverflow.com/questions/70563152

QUESTION

Spring Boot WebClient stops sending requests

Asked 2022-Feb-18 at 14:42

I am running a Spring Boot app that uses WebClient for both non-blocking and blocking HTTP requests. After the app has run for some time, all outgoing HTTP requests seem to get stuck.

WebClient is used to send requests to multiple hosts, but as an example, here is how it is initialized and used to send requests to Telegram:

WebClientConfig:

    @Bean
    public ReactorClientHttpConnector httpClient() {
        HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
                .responseTimeout(Duration.ofMillis(responseTimeout));
        return new ReactorClientHttpConnector(httpClient);
    }

The same ReactorClientHttpConnector is used for all WebClients.

TelegramClient:

    @Autowired
    ReactorClientHttpConnector httpClient;

    WebClient webClient;

    RateLimiter rateLimiter;

    @PostConstruct
    public void init() {
        webClient = WebClient.builder()
                .clientConnector(httpClient)
                .baseUrl(telegramUrl)
                .build();

        rateLimiter = RateLimiter.of("telegram-rate-limiter",
                RateLimiterConfig.custom()
                        .limitRefreshPeriod(Duration.ofMinutes(1))
                        .limitForPeriod(20)
                        .build());
    }

    public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
        webClient.post().uri(String.format("/bot%s/sendMessage", token))
                .contentType(MediaType.APPLICATION_JSON)
                .body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
                        .with("text", message))
                .retrieve()
                .bodyToMono(Void.class)
                .transformDeferred(RateLimiterOperator.of(rateLimiter))
                .block();
    }

The RateLimiter is used to ensure the number of requests does not exceed 20 a minute as specified in the Telegram API.

When the app is started, all requests are resolved normally as expected. But after some time has passed, all requests seem to get stuck. The amount of time needed for this to happen can vary from a few hours to a few days. It happens for all requests to different hosts and is easily noticeable when messages from the TelegramBot stops. Once the requests get stuck, they are stuck indefinitely and I have to restart the app to get it working again.

There are no exceptions in the log that seem to have caused this. Since I maintain a queue for my telegram messages, I can see the point in time when the requests stop when the number of messages in the queue steadily increases and when errors happen in the other processes that are waiting for the requests to resolve.

It does not seem like the requests are even sent out as the connect timeout and response timeout that I have set do not take effect.

I had previously also tried setting idle time to 0 but that did not solve the problem

    @Bean
    public ReactorClientHttpConnector httpClient() {
        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
        HttpClient httpClient = HttpClient.newConnection()
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
                .responseTimeout(Duration.ofMillis(responseTimeout));
        return new ReactorClientHttpConnector(httpClient);
    }

Update:

I enabled metrics and viewed it using micrometer when it got stuck. Interestingly, it shows that there is one connection for Telegram, but also shows no connections on idle, pending or active.

reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0

Could the issue be this missing connection?

Update 2:

I thought this might be related to this other issue: Closing Reactor Netty connection on error status codes

So I updated my HttpClient to this:

    @Bean
    public ReactorClientHttpConnector httpClient() {
        HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
                .doAfterResponseSuccess((r, c) -> c.dispose())
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
                .responseTimeout(Duration.ofMillis(responseTimeout));
        return new ReactorClientHttpConnector(httpClient);
    }

But all that seemed to do is accelerate the occurrence of the problem. Just like before, the active, pending and idle connections do not add up to the total connections. The total is always greater than the other 3 metrics added together.

Update 3: I did a thread dump when the issue happened. There were a total of 74 threads, so I don't think the app is running out of threads.

The dump for the Telegram thread:

"TelegramBot" #20 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=47154.24s tid=0x00007f6b28e73000 nid=0x1c waiting on condition  [0x00007f6aed6fb000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
        - parking to wait for  <0x00000000fa865c80> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1345)
        at java.util.concurrent.CountDownLatch.await(java.base@11.0.13/CountDownLatch.java:232)
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
        at reactor.core.publisher.Mono.block(Mono.java:1707)
        at com.moon.arbitrage.cm.feign.TelegramClient.sendMessage(TelegramClient.java:59)
        at com.moon.arbitrage.cm.service.TelegramService.lambda$sendArbMessage$0(TelegramService.java:53)
        at com.moon.arbitrage.cm.service.TelegramService$$Lambda$1092/0x000000084070f840.run(Unknown Source)
        at com.moon.arbitrage.cm.service.TelegramService.task(TelegramService.java:82)
        at com.moon.arbitrage.cm.service.TelegramService$$Lambda$920/0x0000000840665040.run(Unknown Source)
        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)

   Locked ownable synchronizers:
        - None

The reactor worker threads:

"reactor-http-epoll-1" #15 daemon prio=5 os_prio=0 cpu=810.44ms elapsed=47157.07s tid=0x00007f6b281c4000 nid=0x17 runnable  [0x00007f6b0c46c000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:177)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)

   Locked ownable synchronizers:
        - None

"reactor-http-epoll-2" #16 daemon prio=5 os_prio=0 cpu=1312.16ms elapsed=47157.07s tid=0x00007f6b281c5000 nid=0x18 waiting on condition  [0x00007f6b0c369000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
        - parking to wait for  <0x00000000fa865948> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
        at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
        at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
        at java.util.concurrent.CompletableFuture.get(java.base@11.0.13/CompletableFuture.java:1998)
        at com.moon.arbitrage.cm.service.OrderService.reconcileOrder(OrderService.java:103)
        at com.moon.arbitrage.cm.service.BotService$BotTask.lambda$task$1(BotService.java:383)
        at com.moon.arbitrage.cm.service.BotService$BotTask$$Lambda$1161/0x00000008400af440.accept(Unknown Source)
        at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:171)
        at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
        at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
        at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
        at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
        at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
        at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
        at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
        at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:702)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)

   Locked ownable synchronizers:
        - None

"reactor-http-epoll-3" #17 daemon prio=5 os_prio=0 cpu=171.84ms elapsed=47157.07s tid=0x00007f6b28beb000 nid=0x19 runnable  [0x00007f6b0c26a000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:177)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)

   Locked ownable synchronizers:
        - None

"reactor-http-epoll-4" #18 daemon prio=5 os_prio=0 cpu=188.10ms elapsed=47157.07s tid=0x00007f6b28b7d800 nid=0x1a runnable  [0x00007f6b0c169000]
   java.lang.Thread.State: RUNNABLE
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:177)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)

   Locked ownable synchronizers:
        - None

Seems like one of them is blocked with another task (that isn't even from the Telegram service) but that shouldn't be an issue since the other three worker threads are runnable right?

ANSWER

Answered 2021-Dec-20 at 14:25

I would propose to take a look in the RateLimiter direction. Maybe it does not work as expected, depending on the number of requests your application does over time. From the Javadoc for Ratelimiter: "It is important to note that the number of permits requested never affects the throttling of the request itself ... but it affects the throttling of the next request. I.e., if an expensive task arrives at an idle RateLimiter, it will be granted immediately, but it is the next request that will experience extra throttling, thus paying for the cost of the expensive task." Also helpful might be this discussion: github or github

I could imaginge there is some throttling adding up or other effect in the RateLimiter, i would try to play around with it and make sure this thing really works the way you want. Alternatively, consider using Spring @Scheduled to read from your queue. You might want to spice it up using embedded JMS for further goodies (message persistence etc).

Source https://stackoverflow.com/questions/70357582

QUESTION

How do I get mobile status for discord bot by directly modifying IDENTIFY packet?

Asked 2022-Feb-09 at 15:05

Apparently, discord bots can have mobile status as opposed to the desktop (online) status that one gets by default.

After a bit of digging I found out that such a status is achieved by modifying the IDENTIFY packet in discord.gateway.DiscordWebSocket.identify modifying the value of $browser to Discord Android or Discord iOS should theoretically get us the mobile status.

After modifying code snippets I found online which does this, I end up with this :

def get_mobile():
    """
    The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
    Discord uses that information to know when your phone client and only your phone client has connected to Discord,
    from there they send the extended presence object.
    The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
    the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
    the mobile indicator is not triggered by the mobile client.
    The specific values for the $os, $browser, and $device fields are can change from time to time.
    """
    import ast
    import inspect
    import re
    import discord

    def source(o):
        s = inspect.getsource(o).split("\n")
        indent = len(s[0]) - len(s[0].lstrip())

        return "\n".join(i[indent:] for i in s)

    source_ = source(discord.gateway.DiscordWebSocket.identify)
    patched = re.sub(
        r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
        r"\1Discord Android\2",
        source_,
    )

    loc = {}
    exec(compile(ast.parse(patched), "<string>", "exec"), discord.gateway.__dict__, loc)
    return loc["identify"]

Now all there is left to do is overwrite the discord.gateway.DiscordWebSocket.identify during runtime in the main file, something like this :

import discord
import os
from discord.ext import commands
import mobile_status

discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
bot = commands.Bot(command_prefix="?")

@bot.event
async def on_ready():
    print(f"Sucessfully logged in as {bot.user}")

bot.run(os.getenv("DISCORD_TOKEN"))

And we do get the mobile status successfully

But here's the problem, I wanted to directly modify the file (which held the function) rather than monkey-patching it during runtime. So I cloned the dpy lib locally and edited the file on my machine, it ended up looking like this :

    async def identify(self):
        """Sends the IDENTIFY packet."""
        payload = {
            'op': self.IDENTIFY,
            'd': {
                'token': self.token,
                'properties': {
                    '$os': sys.platform,
                    '$browser': 'Discord Android',
                    '$device': 'Discord Android',
                    '$referrer': '',
                    '$referring_domain': ''
                },
                'compress': True,
                'large_threshold': 250,
                'v': 3
            }
        }
     # ...

(edited both $browser and $device to Discord Android just to be safe)

But this does not work and just gives me the regular desktop online icon.
So the next thing I did is to inspect the identify function after it has been monkey-patched, so I could just look at the source code and see what went wrong earlier, but due to hard luck I got this error :

Traceback (most recent call last):
  File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in <module>
    print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
    lines, lnum = getsourcelines(object)
  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
    lines, lnum = findsource(object)
  File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

Code :

import discord
import os
from discord.ext import commands
import mobile_status
import inspect

discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
bot = commands.Bot(command_prefix="?")

@bot.event
async def on_ready():
    print(f"Sucessfully logged in as {bot.user}")

bot.run(os.getenv("DISCORD_TOKEN"))

Since this same behavior was exhibited for every patched function (aforementioned one and the loc["identify"]) I could no longer use inspect.getsource(...) and then relied upon dis.dis which lead to much more disappointing results

The disassembled data looks exactly identical to the monkey-patched working version, so the directly modified version simply does not work despite function content being the exact same. (In regards to disassembled data)

Notes: Doing Discord iOS directly does not work either, changing the $device to some other value but keeping $browser does not work, I have tried all combinations, none of them work.

TL;DR: How to get mobile status for discord bot without monkey-patching it during runtime?

ANSWER

Answered 2022-Feb-07 at 23:03

The following works by subclassing the relevant class, and duplicating code with the relevant changes. We also have to subclass the Client class, to overwrite the place where the gateway/websocket class is used. This results in a lot of duplicated code, however it does work, and requires neither dirty monkey-patching nor editing the library source code.

However, it does come with many of the same problems as editing the library source code - mainly that as the library is updated, this code will become out of date (if you're using the archived and obsolete version of the library, you have bigger problems instead).

import asyncio
import sys

import aiohttp

import discord
from discord.gateway import DiscordWebSocket, _log
from discord.ext.commands import Bot


class MyGateway(DiscordWebSocket):

    async def identify(self):
        payload = {
            'op': self.IDENTIFY,
            'd': {
                'token': self.token,
                'properties': {
                    '$os': sys.platform,
                    '$browser': 'Discord Android',
                    '$device': 'Discord Android',
                    '$referrer': '',
                    '$referring_domain': ''
                },
                'compress': True,
                'large_threshold': 250,
                'v': 3
            }
        }

        if self.shard_id is not None and self.shard_count is not None:
            payload['d']['shard'] = [self.shard_id, self.shard_count]

        state = self._connection
        if state._activity is not None or state._status is not None:
            payload['d']['presence'] = {
                'status': state._status,
                'game': state._activity,
                'since': 0,
                'afk': False
            }

        if state._intents is not None:
            payload['d']['intents'] = state._intents.value

        await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
        await self.send_as_json(payload)
        _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)


class MyBot(Bot):

    async def connect(self, *, reconnect: bool = True) -> None:
        """|coro|

        Creates a websocket connection and lets the websocket listen
        to messages from Discord. This is a loop that runs the entire
        event system and miscellaneous aspects of the library. Control
        is not resumed until the WebSocket connection is terminated.

        Parameters
        -----------
        reconnect: :class:`bool`
            If we should attempt reconnecting, either due to internet
            failure or a specific failure on Discord's part. Certain
            disconnects that lead to bad state will not be handled (such as
            invalid sharding payloads or bad tokens).

        Raises
        -------
        :exc:`.GatewayNotFound`
            If the gateway to connect to Discord is not found. Usually if this
            is thrown then there is a Discord API outage.
        :exc:`.ConnectionClosed`
            The websocket connection has been terminated.
        """

        backoff = discord.client.ExponentialBackoff()
        ws_params = {
            'initial': True,
            'shard_id': self.shard_id,
        }
        while not self.is_closed():
            try:
                coro = MyGateway.from_client(self, **ws_params)
                self.ws = await asyncio.wait_for(coro, timeout=60.0)
                ws_params['initial'] = False
                while True:
                    await self.ws.poll_event()
            except discord.client.ReconnectWebSocket as e:
                _log.info('Got a request to %s the websocket.', e.op)
                self.dispatch('disconnect')
                ws_params.update(sequence=self.ws.sequence, resume=e.resume, session=self.ws.session_id)
                continue
            except (OSError,
                    discord.HTTPException,
                    discord.GatewayNotFound,
                    discord.ConnectionClosed,
                    aiohttp.ClientError,
                    asyncio.TimeoutError) as exc:

                self.dispatch('disconnect')
                if not reconnect:
                    await self.close()
                    if isinstance(exc, discord.ConnectionClosed) and exc.code == 1000:
                        # clean close, don't re-raise this
                        return
                    raise

                if self.is_closed():
                    return

                # If we get connection reset by peer then try to RESUME
                if isinstance(exc, OSError) and exc.errno in (54, 10054):
                    ws_params.update(sequence=self.ws.sequence, initial=False, resume=True, session=self.ws.session_id)
                    continue

                # We should only get this when an unhandled close code happens,
                # such as a clean disconnect (1000) or a bad state (bad token, no sharding, etc)
                # sometimes, discord sends us 1000 for unknown reasons so we should reconnect
                # regardless and rely on is_closed instead
                if isinstance(exc, discord.ConnectionClosed):
                    if exc.code == 4014:
                        raise discord.PrivilegedIntentsRequired(exc.shard_id) from None
                    if exc.code != 1000:
                        await self.close()
                        raise

                retry = backoff.delay()
                _log.exception("Attempting a reconnect in %.2fs", retry)
                await asyncio.sleep(retry)
                # Always try to RESUME the connection
                # If the connection is not RESUME-able then the gateway will invalidate the session.
                # This is apparently what the official Discord client does.
                ws_params.update(sequence=self.ws.sequence, resume=True, session=self.ws.session_id)


bot = MyBot(command_prefix="?")


@bot.event
async def on_ready():
    print(f"Sucessfully logged in as {bot.user}")

bot.run("YOUR_BOT_TOKEN")

Personally, I think that the following approach, which does include some runtime monkey-patching (but no AST manipulation) is cleaner for this purpose:

import sys
from discord.gateway import DiscordWebSocket, _log
from discord.ext.commands import Bot


async def identify(self):
    payload = {
        'op': self.IDENTIFY,
        'd': {
            'token': self.token,
            'properties': {
                '$os': sys.platform,
                '$browser': 'Discord Android',
                '$device': 'Discord Android',
                '$referrer': '',
                '$referring_domain': ''
            },
            'compress': True,
            'large_threshold': 250,
            'v': 3
        }
    }

    if self.shard_id is not None and self.shard_count is not None:
        payload['d']['shard'] = [self.shard_id, self.shard_count]

    state = self._connection
    if state._activity is not None or state._status is not None:
        payload['d']['presence'] = {
            'status': state._status,
            'game': state._activity,
            'since': 0,
            'afk': False
        }

    if state._intents is not None:
        payload['d']['intents'] = state._intents.value

    await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
    await self.send_as_json(payload)
    _log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)


DiscordWebSocket.identify = identify
bot = Bot(command_prefix="?")


@bot.event
async def on_ready():
    print(f"Sucessfully logged in as {bot.user}")

bot.run("YOUR_DISCORD_TOKEN")

As to why editing the library source code did not work for you, I can only assume that you have edited the wrong copy of the file, as people have commented.

Source https://stackoverflow.com/questions/70903401

QUESTION

Changing Category/Channels Permissions Returns Error "Missing Permissions" - Novus/Discord.py

Asked 2022-Feb-08 at 01:39

SOLVED - The categories/channels were set to private which is why they couldn't be changed even with the correct permissions. My solution was to tell the user which ones and to add the role with the permissions manually. I added a message that said they can just give the bot administrator permissions and it will do it for them. Hopefully, this helps anyone who runs into this in the future.

I'm trying to change the permissions of multiple categories and channels using the following code:

role = get(guild.roles, name="Fun")
for channel in guild.channels:
    overwrites = channel.overwrites_for(role)
    overwrites.view_channel = False
    await channel.set_permissions(role, overwrite=overwrites)

I get the following error:

await channel.set_permissions(role, overwrite=overwrites)
discord.errors.Forbidden: 403 Forbidden (error code: 50013): Missing Permissions

It will work if I give the bot the administrator permission. However, it won't work if I give the bot every single other permission, the bot is at the top of the roles, and it has every category permission. What am I missing? I'm using Novus (fork of discord.py so it's pretty much the same thing).

Edit: I've asked people in the discord.py, Novus, and discord developers discord servers and they can't help me either. With the exception of the discord developers server they just didn't answer my question.

Edit 2: I was able to get it working when I manually added the bot's role to the category and enabled view_channel. This won't work, however, because that means the server owner has to manually do this for every category and channel which is really inconvenient.

ANSWER

Answered 2021-Oct-31 at 12:05

This is simply how Discord works, if a channel is private and the bot does not have administrator permissions or role that has permissions to view the channel, it is not able to view it.

You can create a new category that the bot can access like this:

role = await guild.create_role(name="Fun")
category = await guild.create_category("category_name")
await category.set_permissions(role, read_messages=True, send_messages=True, connect=True, speak=True)
await category.set_permissions(ctx.guild.me, read_messages=True, send_messages=True, speak=True)

(How to give own bot permission to see categories?)

Source https://stackoverflow.com/questions/69694159

QUESTION

Error [ERR_REQUIRE_ESM]: require() of ES Module not supported

Asked 2022-Feb-03 at 22:08

I'm trying to make a Discord bot that just says if someone is online on the game.

However I keep getting this message:

[ERR_REQUIRE_ESM]: require() of ES Module from not supported. Instead change the require of index.js in... to a dynamic import() which is available in all CommonJS modules.

This is my code:

    module.exports = {
        name: 'username',
        description: "this is the username command",
        async execute(message, args) {

            const fetch = require('node-fetch');

            if (args.length !== 1) {
                return message.channel.send("invalid username wtf")
            }

            const ign = args[0]

            if (ign.length > 16 || ign.length < 3) {
                return message.channel.send("invalid username wtf")
            }

            const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
            const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());

            if (uuid.length !== 32) {
                return;
            }

            if (onlineinfo.success) {
                if (onlineinfo.session.online) {
                    message.channel.send("they are online")
                }
                else {
                    message.channel.send("they are offline")
                }
            }
            else {
                message.channel.send("hypixel api bad wtf")
            }
        }
    }

This is my package.json file:

{
    "name": "discordbot",
    "version": "1.0.0",
    "main": "main.js",
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1",
        "start": "node main.js"
    },
    "author": "",
    "license": "ISC",
    "description": "",
    "dependencies": {
        "discord.js": "^13.0.1",
        "node-fetch": "^3.0.0"
    }
}

ANSWER

Answered 2021-Sep-07 at 06:38

node-fetch v3 recently stopped support for the require way of importing it in favor of ES Modules. You'll need to use ESM imports now, like:

import fetch from "node-fetch";

at the top of your file.

Source https://stackoverflow.com/questions/69081410

QUESTION

How to check if a bot can DM a user

Asked 2022-Jan-22 at 22:03

If a user has the privacy setting "Allow direct messages from server members" turned off and a discord bot calls

await user.dm_channel.send("Hello there")

You'll get this error:

discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user

I would like to check whether I can message a user without sending them a message. Trying to send a message and catching this error does not work for me, because I don't want a message to get sent in the event that the bot is allowed to message.

I have tried this:

print(user.dm_channel.permissions_for(bot).send_messages)

but it always returns True, even if the message is not permitted.

I have also tried this:

channel = await user.create_dm()
if channel is None:
    ...

but unfortunately, it seems that "has permission to message user" and "has permission to create a dm channel" are considered different.

EDIT

To clarify the exact usage since there seems to be a bit of confusion, take this example. There is a server, and 3 users in question: Me, My Bot, and Steve. Steve has "Allow direct messages from server members" checked off.

The bot has a command called !newgame which accepts a list of users and starts a game amongst them, which involves DMing some of the members of the game. Because of Steve's privacy settings, he cannot play the game (since the bot will need to message him). If I do

!newgame @DJMcMayhem @Steve

I'd like to provide a response like:

> I can't start a game with that list of users because @Steve has the wrong privacy settings.

But as far as I know right now, the only way to find out if Steve can play is by first attempting to message every user, which I'd like to avoid.

ANSWER

Answered 2022-Jan-22 at 22:03

Explanation

You can generate a Bad Request to the dm_channel. This can be accomplished by setting content to None, for example.

If it returns with 400 Bad Request, you can DM them. If it returns with 403 Forbidden, you can't.

Code

async def can_dm_user(user: discord.User) -> bool:
    ch = user.dm_channel
    if ch is None:
        ch = await user.create_dm()

    try:
        await ch.send()
    except discord.Forbidden:
        return False
    except discord.HTTPException:
        return True

Source https://stackoverflow.com/questions/70660854

QUESTION

How can I send Dynamic website content to scrapy with the html content generated by selenium browser?

Asked 2022-Jan-20 at 15:35

I am working on certain stock-related projects where I have had a task to scrape all data on a daily basis for the last 5 years. i.e from 2016 to date. I particularly thought of using selenium because I can use crawler and bot to scrape the data based on the date. So I used the use of button click with selenium and now I want the same data that is displayed by the selenium browser to be fed by scrappy. This is the website I am working on right now. I have written the following code inside scrappy spider.

class FloorSheetSpider(scrapy.Spider):
    name = "nepse"

    def start_requests(self):

        driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
        
     
        floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']

        for date in floorsheet_dates:
            driver.get(
                "https://merolagani.com/Floorsheet.aspx")

            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
                                ).send_keys(date)
            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
            total_length = driver.find_element(By.XPATH,
                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
            z = int((total_length.split()[-1]).replace(']', ''))    
            for data in range(z, z + 1):
                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
                self.url = driver.page_source
                yield Request(url=self.url, callback=self.parse)

               
    def parse(self, response, **kwargs):
        for value in response.xpath('//tbody/tr'):
            print(value.css('td::text').extract()[1])
            print("ok"*200)

Update: Error after answer is

2022-01-14 14:11:36 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
    start_requests = iter(self.spider.start_requests())
TypeError: 'NoneType' object is not iterable

I want to send current web html content to scrapy feeder but I am getting unusal error for past 2 days any help or suggestions will be very much appreciated.

ANSWER

Answered 2022-Jan-14 at 09:30

The 2 solutions are not very different. Solution #2 fits better to your question, but choose whatever you prefer.

Solution 1 - create a response with the html's body from the driver and scraping it right away (you can also pass it as an argument to a function):

import scrapy
from selenium import webdriver
from selenium.webdriver.common.by import By
from scrapy.http import HtmlResponse


class FloorSheetSpider(scrapy.Spider):
    name = "nepse"

    def start_requests(self):

        # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
        driver = webdriver.Chrome()

        floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']

        for date in floorsheet_dates:
            driver.get(
                "https://merolagani.com/Floorsheet.aspx")

            driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
                                ).send_keys(date)
            driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
            total_length = driver.find_element(By.XPATH,
                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
            z = int((total_length.split()[-1]).replace(']', ''))
            for data in range(1, z + 1):
                driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
                self.body = driver.page_source

                response = HtmlResponse(url=driver.current_url, body=self.body, encoding='utf-8')
                for value in response.xpath('//tbody/tr'):
                    print(value.css('td::text').extract()[1])
                    print("ok"*200)

        # return an empty requests list
        return []

Solution 2 - with super simple downloader middleware:

(You might have a delay here in parse method so be patient).

import scrapy
from scrapy import Request
from scrapy.http import HtmlResponse
from selenium import webdriver
from selenium.webdriver.common.by import By


class SeleniumMiddleware(object):
    def process_request(self, request, spider):
        url = spider.driver.current_url
        body = spider.driver.page_source
        return HtmlResponse(url=url, body=body, encoding='utf-8', request=request)


class FloorSheetSpider(scrapy.Spider):
    name = "nepse"

    custom_settings = {
        'DOWNLOADER_MIDDLEWARES': {
            'tempbuffer.spiders.yetanotherspider.SeleniumMiddleware': 543,
            # 'projects_name.path.to.your.pipeline': 543
        }
    }
    driver = webdriver.Chrome()

    def start_requests(self):

        # driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())


        floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']

        for date in floorsheet_dates:
            self.driver.get(
                "https://merolagani.com/Floorsheet.aspx")

            self.driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
                                ).send_keys(date)
            self.driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
            total_length = self.driver.find_element(By.XPATH,
                                               "//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
            z = int((total_length.split()[-1]).replace(']', ''))
            for data in range(1, z + 1):
                self.driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
                self.body = self.driver.page_source
                self.url = self.driver.current_url

                yield Request(url=self.url, callback=self.parse, dont_filter=True)

    def parse(self, response, **kwargs):
        print('test ok')
        for value in response.xpath('//tbody/tr'):
            print(value.css('td::text').extract()[1])
            print("ok"*200)

Notice that I've used chrome so change it back to firefox like in your original code.

Source https://stackoverflow.com/questions/70651053

QUESTION

Python/Selenium web scrap how to find hidden src value from a links?

Asked 2022-Jan-16 at 02:28

Scrapping links should be a simple feat, usually just grabbing the src value of the a tag.

I recently came across this website (https://sunteccity.com.sg/promotions) where the href value of a tags of each item cannot be found, but the redirection still works. I'm trying to figure out a way to grab the items and their corresponding links. My typical python selenium code looks something as such

all_items = bot.find_elements_by_class_name('thumb-img')
for promo in all_items:
    a = promo.find_elements_by_tag_name("a")
    print("a[0]: ", a[0].get_attribute("href"))

However, I can't seem to retrieve any href, onclick attributes, and I'm wondering if this is even possible. I noticed that I couldn't do a right-click, open link in new tab as well.

Are there any ways around getting the links of all these items?

Edit: Are there any ways to retrieve all the links of the items on the pages?

i.e.

https://sunteccity.com.sg/promotions/724
https://sunteccity.com.sg/promotions/731
https://sunteccity.com.sg/promotions/751
https://sunteccity.com.sg/promotions/752
https://sunteccity.com.sg/promotions/754
https://sunteccity.com.sg/promotions/280
...

Edit: Adding an image of one such anchor tag for better clarity:

ANSWER

Answered 2022-Jan-15 at 19:47

You are using a wrong locator. It brings you a lot of irrelevant elements.
Instead of find_elements_by_class_name('thumb-img') please try find_elements_by_css_selector('.collections-page .thumb-img') so your code will be

all_items = bot.find_elements_by_css_selector('.collections-page .thumb-img')
for promo in all_items:
    a = promo.find_elements_by_tag_name("a")
    print("a[0]: ", a[0].get_attribute("href"))

You can also get the desired links directly by .collections-page .thumb-img a locator so that your code could be:

links = bot.find_elements_by_css_selector('.collections-page .thumb-img a')
for link in links:
    print(link.get_attribute("href"))

Source https://stackoverflow.com/questions/70721360

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install python-aria-mirror-bot

Fill up rest of the fields. Meaning of each fields are discussed below:. Note: You can limit maximum concurrent downloads by changing the value of MAX_CONCURRENT_DOWNLOADS in aria.sh. By default, it's set to 2.
Remove the first line saying:
BOT_TOKEN : The telegram bot token that you get from @BotFather
GDRIVE_FOLDER_ID : This is the folder ID of the Google Drive Folder to which you want to upload all the mirrors.
DOWNLOAD_DIR : The path to the local folder where the downloads should be downloaded to
DOWNLOAD_STATUS_UPDATE_INTERVAL : A short interval of time in seconds after which the Mirror progress message is updated. (I recommend to keep it 5 seconds at least)
OWNER_ID : The Telegram user ID (not username) of the owner of the bot
AUTO_DELETE_MESSAGE_DURATION : Interval of time (in seconds), after which the bot deletes it's message (and command message) which is expected to be viewed instantly. Note: Set to -1 to never automatically delete messages
IS_TEAM_DRIVE : (Optional field) Set to "True" if GDRIVE_FOLDER_ID is from a Team Drive else False or Leave it empty.
USE_SERVICE_ACCOUNTS: (Optional field) (Leave empty if unsure) Whether to use service accounts or not. For this to work see "Using service accounts" section below.
INDEX_URL : (Optional field) Refer to https://github.com/maple3142/GDIndex/ The URL should not have any trailing '/'
TELEGRAM_API : This is to authenticate to your telegram account for downloading Telegram files. You can get this from https://my.telegram.org DO NOT put this in quotes.
TELEGRAM_HASH : This is to authenticate to your telegram account for downloading Telegram files. You can get this from https://my.telegram.org
USER_SESSION_STRING : Session string generated by running:
MEGA_KEY: Mega.nz api key to mirror mega.nz links. Get it from Mega SDK Page
MEGA_USERNAME: Your mega email id (You can leave it empty, it will start megasdkrest server in anonymous mode)
MEGA_PASSWORD: Your password for your mega.nz account. (NOTE: You must deactivate 2FA to use the account with the bot otherwise bot will not be able to sign in)

Support

Mirroring direct download links to google driveDownload progressUpload progressDownload/upload speeds and ETAsDocker supportUploading To Team Drives.Index Link supportService account supportMirror all youtube-dl supported linksMirror telegram filesStable Mega.nz support