InstaPy | 📷 Instagram Bot - Tool for automated Instagram interactions | Bot library
kandi X-RAY | InstaPy Summary
Support
Quality
Security
License
Reuse
- Unfollow a user .
- Validate a given username .
- Interactively interactively .
- Login a user .
- Get all followers of a user .
- Get following data for a given username .
- Get the number of active users .
- Extract information from Instagram .
- Analyze text .
- Gets a list of users through GraphQL .
InstaPy Key Features
InstaPy Examples and Code Snippets
Lines of Code : 30License : Non-SPDX (NOASSERTION)
Copy
. ├── CHANGELOG.md ├── CONTRIBUTING.md ├── instapy_bot │ ├── bot │ │ ├── __init__.py │ │ ├── logger │ │ │ ├── __init__.py │ │ │ └── logger.py │ │ ├── mailer │ │ │ ├── __init__.py │ │ │ └── mailer.py │ │ ├── __main__.py │ │ └── utils │ │ ├── __init__.py │ │ └── photo.py │ │ │ ├── cli │ │ ├── cli.py │ │ ├── __init__.py │ │ ├── __main__.py │ │ ├── media.py │ │ └── session.py │ ├── __init__.py │ └── version.py ├── LICENSE ├── README.md ├── requirements.txt └── setup.py
$ instapy-bot -f ./photos -t 3600
$ instapy-bot -c /path/to/config
[credentials] username = instagram_username password = instagram_password [config] timeout = 3600 folder = /path/to/photos/folder [mailer] to = mail_to@example.com username = mailer_username password = mailer_password [caption] text = Photo of the day reg = candid nature photooftheday bnw = bnw blackandwhite
License : No LicenseCopy
$ heroku login
$ heroku container:login
$ ./heroku_init.sh
docker-compose build
$ heroku container:push instapy && heroku container:release instapy
$ heroku ps:scale instapy=1
$ heroku logs --tailTrending Discussions on Bot
Trending Discussions on Bot
QUESTION
It's my first time seeing this feature from a Discord bot. I tried looking everywhere but it seems that I have failed. There's this feature from Captcha.bot Discord bot where you can accept input from a pop-up window inside Discord.
There's a button in an embedded message made by Captcha.bot where you will have to answer a Captcha test. After pressing the button, it creates a pop-up window like this.
After placing the right answer on the captcha bot, here's the aftermath of the experience.
All I want to learn is how to summon that pop-up window using Discord.js if it's even possible or at least learn how they did it.
ANSWER
Answered 2022-Mar-30 at 07:12Those are called modals, and they will be available in the next discord.js version, v14. There is already a pull request for this.
In the meantime, you can use an npm package like discord-modals or discordjs-modal.
You can find a working example with the discord-modals package below. Don't forget to install it first using npm i discord-modals.
const {
Client,
Intents,
MessageActionRow,
MessageButton,
} = require('discord.js');
const discordModals = require('discord-modals');
const { Modal, TextInputComponent, showModal } = discordModals;
const TOKEN = 'YOUR TOKEN HERE';
const client = new Client({
intents: [Intents.FLAGS.GUILDS, Intents.FLAGS.GUILD_MESSAGES],
});
discordModals(client);
client.on('messageCreate', (message) => {
if (message.author.bot) return;
let button = new MessageActionRow();
button.addComponents(
new MessageButton()
.setCustomId('verification-button')
.setStyle('PRIMARY')
.setLabel('Open modal dialog'),
);
message.reply({
components: [button],
});
});
client.on('interactionCreate', async (interaction) => {
if (interaction.isButton()) {
if (interaction.customId === 'verification-button') {
const modal = new Modal() // We create a Modal
.setCustomId('verification-modal')
.setTitle('Verify yourself')
.addComponents([
new TextInputComponent()
.setCustomId('verification-input')
.setLabel('Answer')
.setStyle('SHORT')
.setMinLength(4)
.setMaxLength(12)
.setPlaceholder('ABCDEF')
.setRequired(true),
]);
showModal(modal, {
client,
interaction,
});
}
}
});
client.on('modalSubmit', async (modal) => {
if (modal.customId === 'verification-modal') {
const response = modal.getTextInputValue('verification-input');
modal.reply(`Yay, your answer is submitted: "${response}"`);
}
});
client.once('ready', () => {
console.log('Bot v13 is connected...');
});
client.login(TOKEN);
QUESTION
Using AWS Lambda functions with Python and Selenium, I want to create a undetectable headless chrome scraper by passing a headless chrome test. I check the undetectability of my headless scraper by opening up the test and taking a screenshot. I ran this test on a Local IDE and on a Lambda server.
Implementation:I will be using a python library called selenium-stealth and will follow their basic configuration:
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
)
I implemented this configuration on a Local IDE as well as an AWS Lambda Server to compare the results.
Local IDE:Found below are the test results running on a local IDE:
Lambda Server:When I run this on a Lambda server, both the WebGL Vendor and Renderer are blank. as shown below:
I even tried to manually change the WebGL Vendor/Renderer using the following JavaScript command:
driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {"source": "WebGLRenderingContext.prototype.getParameter = function(parameter) {if (parameter === 37445) {return 'VENDOR_INPUT';}if (parameter === 37446) {return 'RENDERER_INPUT';}return getParameter(parameter);};"})
Then I thought maybe that it could be something wrong with the parameter number. I configured the command execution without the if statement, but the same thing happened: It worked on my Local IDE but had no effect on an AWS Lambda Server.
Simply Put:Is it possible to add Vendor/Renderer on AWS Lambda? In my efforts, it seems that there is no possible way. I made sure to submit this issue on the selenium-stealth GitHub Repository.
ANSWER
Answered 2021-Dec-18 at 02:01WebGL is a cross-platform, open web standard for a low-level 3D graphics API based on OpenGL ES, exposed to ECMAScript via the HTML5 Canvas element. WebGL at it's core is a Shader-based API using GLSL, with constructs that are semantically similar to those of the underlying OpenGL ES API. It follows the OpenGL ES specification, with some exceptions for the out of memory-managed languages such as JavaScript. WebGL 1.0 exposes the OpenGL ES 2.0 feature set; WebGL 2.0 exposes the OpenGL ES 3.0 API.
Now, with the availability of Selenium Stealth building of Undetectable Scraper using Selenium driven ChromeDriver initiated google-chrome Browsing Context have become much more easier.
selenium-stealthselenium-stealth is a python package selenium-stealth to prevent detection. This programme tries to make python selenium more stealthy. However, as of now selenium-stealth only support Selenium Chrome.
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium_stealth import stealth
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
# Selenium Stealth settings
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
)
driver.get("https://bot.sannysoft.com/")
Browser Screenshot:
Changing WebGL Vendor/Renderer in AWS LambdaYou can find a detailed relevant discussion in Can a website detect when you are using Selenium with chromedriver?
AWS Lambda enables us to deliver compressed WebGL websites to end users. When requested webpage objects are compressed, the transfer size is reduced, leading to faster downloads, lower cloud storage fees, and lower data transfer fees. Improved load times also directly influence the viewer experience and retention, which helps in improving website conversion and discoverability. Using WebGL, websites are more immersive while still being accessible via a browser URL. Through this technique AWS Lambda to automatically compress the objects uploaded to S3.
Background on compression and WebGLHTTP compression is a capability that can be built into web servers and web clients to improve transfer speed and bandwidth utilization. This capability is negotiated between the server and the client using an HTTP header which may indicate that a resource being transferred, cached, or otherwise referenced is compressed. AWS Lambda on the server-side supports Content-Encoding header.
On the client-side, most browsers today support brotli and gzip compression through HTTP headers (Accept-Encoding: deflate, br, gzip) and can handle server response headers. This means browsers will automatically download and decompress content from a web server at the client-side, before rendering webpages to the viewer.
Due to this constraint you may not be able to change the WebGL Vendor/Renderer in AWS Lambda, else it may directly affect the process of rendering webpages to the viewers and can stand out to be a bottleneck in UX.
tl; drYou can find a couple of relevant detailed discussion in:
QUESTION
I handle a channelDelete event in my discord bot. My original intent was to do the following:
- Listen for when a channel is deleted
- Check to see if its type equals 'GUILD_CATEGORY'
- Delete all the channels under that category
I can typically access channels under a CategoryChannel through its property called children anywhere else except during this event...
module.exports = {
name: 'channelDelete',
once: false,
async execute(channel) {
if(channel.type == 'GUILD_CATEGORY')
for(let [child_id, child] of channel.children)
if (child.deletable) child.delete();
},
}
I can confirm the code is being executed, but the problem is that the incoming channel object is in a state where it is already deleted, and I cannot get the children:
- During debugging, I noticed the channel object has the property
deleted: true - The
childrenproperty is empty, even though I know there were channels in that channel category prior to deletion
Is there a way for my bot to collect and handle the children of a CategoryChannel prior to its deletion?
ANSWER
Answered 2022-Feb-19 at 14:09Unfortunately, this is how CategoryChannels work in discord.js...
When the category is deleted, discord.js sends a request to the API to delete the channel. Only then, Discord sends you the event after the category is deleted.
What happens next is that the children are not located in the category anymore! So you will not be able to get the children inside the CategoryChannel object.
This is the code for the children property
get children() {
return this.guild.channels.cache.filter(c => c.parentId === this.id);
}
It filters the channels which are in the guild, but those children channels are not located in the category anymore. Which is the reason the property is empty.
The (Only?) SolutionThis will require caching the channels themselves. As there aren't really any other solutions.
You may cache the channels in different ways.
Just remember... there are instances and references in Javascript, and failure to acknowledge may lead to weird behaviors. The following code only works for small bots without sharding, just so you know.
const cachedChannels = new Map()
client.on('ready', () => {
// Looping thru guilds...
client.guilds.forEach((guild) => {
// Filtering the channels to category channels only and looping thru them...
guild.channels.filter((channel) => channel.type === "GUILD_CATEGORY").forEach((category) => {
// Note on references and instances: Numbers, Strings and Booleans do not have instances like Object, Arrays, etc. do
// We are using an array here to abuse the references
cachedChannels.set(category.id, [])
// Looping thru the children channels...
category.children.forEach((children) => {
// This is the array we've created earlier
const storedChannels = cachedChannels.get(category.id)
// Pushing the ID so we can fetch them later
storedChannels.push(children.id)
})
})
})
})
client.on('channelDelete', (channel) => {
if (channel.type === "GUILD_CATEGORY") {
// Looping thru our cached channel IDs defined earlier
cachedChannels.get(channel.id).forEach((id) => {
const child = channel.guild.channels.get(id)
child.delete()
})
// Delete the cached IDs to save memory
cachedChannels.delete(channel.id)
}
})
QUESTION
I am running a Spring Boot app that uses WebClient for both non-blocking and blocking HTTP requests. After the app has run for some time, all outgoing HTTP requests seem to get stuck.
WebClient is used to send requests to multiple hosts, but as an example, here is how it is initialized and used to send requests to Telegram:
WebClientConfig:
@Bean
public ReactorClientHttpConnector httpClient() {
HttpClient.create(ConnectionProvider.builder("connectionProvider").build())
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
.responseTimeout(Duration.ofMillis(responseTimeout));
return new ReactorClientHttpConnector(httpClient);
}
The same ReactorClientHttpConnector is used for all WebClients.
TelegramClient:
@Autowired
ReactorClientHttpConnector httpClient;
WebClient webClient;
RateLimiter rateLimiter;
@PostConstruct
public void init() {
webClient = WebClient.builder()
.clientConnector(httpClient)
.baseUrl(telegramUrl)
.build();
rateLimiter = RateLimiter.of("telegram-rate-limiter",
RateLimiterConfig.custom()
.limitRefreshPeriod(Duration.ofMinutes(1))
.limitForPeriod(20)
.build());
}
public void sendMessage(@PathVariable("token") String token, @RequestParam("chat_id") long chatId, @RequestParam("text") String message) {
webClient.post().uri(String.format("/bot%s/sendMessage", token))
.contentType(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromFormData("chat_id", String.valueOf(chatId))
.with("text", message))
.retrieve()
.bodyToMono(Void.class)
.transformDeferred(RateLimiterOperator.of(rateLimiter))
.block();
}
The RateLimiter is used to ensure the number of requests does not exceed 20 a minute as specified in the Telegram API.
When the app is started, all requests are resolved normally as expected. But after some time has passed, all requests seem to get stuck. The amount of time needed for this to happen can vary from a few hours to a few days. It happens for all requests to different hosts and is easily noticeable when messages from the TelegramBot stops. Once the requests get stuck, they are stuck indefinitely and I have to restart the app to get it working again.
There are no exceptions in the log that seem to have caused this. Since I maintain a queue for my telegram messages, I can see the point in time when the requests stop when the number of messages in the queue steadily increases and when errors happen in the other processes that are waiting for the requests to resolve.
It does not seem like the requests are even sent out as the connect timeout and response timeout that I have set do not take effect.
I had previously also tried setting idle time to 0 but that did not solve the problem
@Bean
public ReactorClientHttpConnector httpClient() {
HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").maxConnections(1000).maxIdleTime(Duration.ofSeconds(0)).build())
HttpClient httpClient = HttpClient.newConnection()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
.responseTimeout(Duration.ofMillis(responseTimeout));
return new ReactorClientHttpConnector(httpClient);
}
Update:
I enabled metrics and viewed it using micrometer when it got stuck. Interestingly, it shows that there is one connection for Telegram, but also shows no connections on idle, pending or active.
reactor_netty_connection_provider_idle_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
reactor_netty_connection_provider_pending_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
reactor_netty_connection_provider_active_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 0.0
reactor_netty_connection_provider_total_connections{id="-1268283746",name="connectionProvider",remote_address="api.telegram.org:443",} 1.0
Could the issue be this missing connection?
Update 2:
I thought this might be related to this other issue: Closing Reactor Netty connection on error status codes
So I updated my HttpClient to this:
@Bean
public ReactorClientHttpConnector httpClient() {
HttpClient httpClient = HttpClient.create(ConnectionProvider.builder("connectionProvider").metrics(true).build())
.doAfterResponseSuccess((r, c) -> c.dispose())
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, connectTimeout)
.responseTimeout(Duration.ofMillis(responseTimeout));
return new ReactorClientHttpConnector(httpClient);
}
But all that seemed to do is accelerate the occurrence of the problem. Just like before, the active, pending and idle connections do not add up to the total connections. The total is always greater than the other 3 metrics added together.
Update 3: I did a thread dump when the issue happened. There were a total of 74 threads, so I don't think the app is running out of threads.
The dump for the Telegram thread:
"TelegramBot" #20 daemon prio=5 os_prio=0 cpu=14.65ms elapsed=47154.24s tid=0x00007f6b28e73000 nid=0x1c waiting on condition [0x00007f6aed6fb000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x00000000fa865c80> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.13/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1039)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.13/AbstractQueuedSynchronizer.java:1345)
at java.util.concurrent.CountDownLatch.await(java.base@11.0.13/CountDownLatch.java:232)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
at reactor.core.publisher.Mono.block(Mono.java:1707)
at com.moon.arbitrage.cm.feign.TelegramClient.sendMessage(TelegramClient.java:59)
at com.moon.arbitrage.cm.service.TelegramService.lambda$sendArbMessage$0(TelegramService.java:53)
at com.moon.arbitrage.cm.service.TelegramService$$Lambda$1092/0x000000084070f840.run(Unknown Source)
at com.moon.arbitrage.cm.service.TelegramService.task(TelegramService.java:82)
at com.moon.arbitrage.cm.service.TelegramService$$Lambda$920/0x0000000840665040.run(Unknown Source)
at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
Locked ownable synchronizers:
- None
The reactor worker threads:
"reactor-http-epoll-1" #15 daemon prio=5 os_prio=0 cpu=810.44ms elapsed=47157.07s tid=0x00007f6b281c4000 nid=0x17 runnable [0x00007f6b0c46c000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:177)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
Locked ownable synchronizers:
- None
"reactor-http-epoll-2" #16 daemon prio=5 os_prio=0 cpu=1312.16ms elapsed=47157.07s tid=0x00007f6b281c5000 nid=0x18 waiting on condition [0x00007f6b0c369000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.13/Native Method)
- parking to wait for <0x00000000fa865948> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.13/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.13/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.13/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.13/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.get(java.base@11.0.13/CompletableFuture.java:1998)
at com.moon.arbitrage.cm.service.OrderService.reconcileOrder(OrderService.java:103)
at com.moon.arbitrage.cm.service.BotService$BotTask.lambda$task$1(BotService.java:383)
at com.moon.arbitrage.cm.service.BotService$BotTask$$Lambda$1161/0x00000008400af440.accept(Unknown Source)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:171)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400)
at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419)
at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:473)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:702)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
Locked ownable synchronizers:
- None
"reactor-http-epoll-3" #17 daemon prio=5 os_prio=0 cpu=171.84ms elapsed=47157.07s tid=0x00007f6b28beb000 nid=0x19 runnable [0x00007f6b0c26a000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:177)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
Locked ownable synchronizers:
- None
"reactor-http-epoll-4" #18 daemon prio=5 os_prio=0 cpu=188.10ms elapsed=47157.07s tid=0x00007f6b28b7d800 nid=0x1a runnable [0x00007f6b0c169000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:177)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(java.base@11.0.13/Thread.java:829)
Locked ownable synchronizers:
- None
Seems like one of them is blocked with another task (that isn't even from the Telegram service) but that shouldn't be an issue since the other three worker threads are runnable right?
ANSWER
Answered 2021-Dec-20 at 14:25I would propose to take a look in the RateLimiter direction. Maybe it does not work as expected, depending on the number of requests your application does over time. From the Javadoc for Ratelimiter: "It is important to note that the number of permits requested never affects the throttling of the request itself ... but it affects the throttling of the next request. I.e., if an expensive task arrives at an idle RateLimiter, it will be granted immediately, but it is the next request that will experience extra throttling, thus paying for the cost of the expensive task." Also helpful might be this discussion: github or github
I could imaginge there is some throttling adding up or other effect in the RateLimiter, i would try to play around with it and make sure this thing really works the way you want. Alternatively, consider using Spring @Scheduled to read from your queue. You might want to spice it up using embedded JMS for further goodies (message persistence etc).
QUESTION
Apparently, discord bots can have mobile status as opposed to the desktop (online) status that one gets by default.
After a bit of digging I found out that such a status is achieved by modifying the IDENTIFY packet in discord.gateway.DiscordWebSocket.identify modifying the value of $browser to Discord Android or Discord iOS should theoretically get us the mobile status.
After modifying code snippets I found online which does this, I end up with this :
def get_mobile():
"""
The Gateway's IDENTIFY packet contains a properties field, containing $os, $browser and $device fields.
Discord uses that information to know when your phone client and only your phone client has connected to Discord,
from there they send the extended presence object.
The exact field that is checked is the $browser field. If it's set to Discord Android on desktop,
the mobile indicator is is triggered by the desktop client. If it's set to Discord Client on mobile,
the mobile indicator is not triggered by the mobile client.
The specific values for the $os, $browser, and $device fields are can change from time to time.
"""
import ast
import inspect
import re
import discord
def source(o):
s = inspect.getsource(o).split("\n")
indent = len(s[0]) - len(s[0].lstrip())
return "\n".join(i[indent:] for i in s)
source_ = source(discord.gateway.DiscordWebSocket.identify)
patched = re.sub(
r'([\'"]\$browser[\'"]:\s?[\'"]).+([\'"])',
r"\1Discord Android\2",
source_,
)
loc = {}
exec(compile(ast.parse(patched), "", "exec"), discord.gateway.__dict__, loc)
return loc["identify"]
Now all there is left to do is overwrite the discord.gateway.DiscordWebSocket.identify during runtime in the main file, something like this :
import discord
import os
from discord.ext import commands
import mobile_status
discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
bot = commands.Bot(command_prefix="?")
@bot.event
async def on_ready():
print(f"Sucessfully logged in as {bot.user}")
bot.run(os.getenv("DISCORD_TOKEN"))
And we do get the mobile status successfully
But here's the problem, I wanted to directly modify the file (which held the function) rather than monkey-patching it during runtime. So I cloned the dpy lib locally and edited the file on my machine, it ended up looking like this :
async def identify(self):
"""Sends the IDENTIFY packet."""
payload = {
'op': self.IDENTIFY,
'd': {
'token': self.token,
'properties': {
'$os': sys.platform,
'$browser': 'Discord Android',
'$device': 'Discord Android',
'$referrer': '',
'$referring_domain': ''
},
'compress': True,
'large_threshold': 250,
'v': 3
}
}
# ...
(edited both $browser and $device to Discord Android just to be safe)
But this does not work and just gives me the regular desktop online icon.
So the next thing I did is to inspect the identify function after it has been monkey-patched, so I could just look at the source code and see what went wrong earlier, but due to hard luck I got this error :
Traceback (most recent call last):
File "c:\Users\Achxy\Desktop\fresh\file.py", line 8, in
print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1024, in getsource
lines, lnum = getsourcelines(object)
File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 1006, in getsourcelines
lines, lnum = findsource(object)
File "C:\Users\Achxy\AppData\Local\Programs\Python\Python39\lib\inspect.py", line 835, in findsource
raise OSError('could not get source code')
OSError: could not get source code
Code :
import discord
import os
from discord.ext import commands
import mobile_status
import inspect
discord.gateway.DiscordWebSocket.identify = mobile_status.get_mobile()
print(inspect.getsource(discord.gateway.DiscordWebSocket.identify))
bot = commands.Bot(command_prefix="?")
@bot.event
async def on_ready():
print(f"Sucessfully logged in as {bot.user}")
bot.run(os.getenv("DISCORD_TOKEN"))
Since this same behavior was exhibited for every patched function (aforementioned one and the loc["identify"]) I could no longer use inspect.getsource(...) and then relied upon dis.dis which lead to much more disappointing results
The disassembled data looks exactly identical to the monkey-patched working version, so the directly modified version simply does not work despite function content being the exact same. (In regards to disassembled data)
Notes: Doing Discord iOS directly does not work either, changing the $device to some other value but keeping $browser does not work, I have tried all combinations, none of them work.
TL;DR: How to get mobile status for discord bot without monkey-patching it during runtime?
ANSWER
Answered 2022-Feb-07 at 23:03The following works by subclassing the relevant class, and duplicating code with the relevant changes. We also have to subclass the Client class, to overwrite the place where the gateway/websocket class is used. This results in a lot of duplicated code, however it does work, and requires neither dirty monkey-patching nor editing the library source code.
However, it does come with many of the same problems as editing the library source code - mainly that as the library is updated, this code will become out of date (if you're using the archived and obsolete version of the library, you have bigger problems instead).
import asyncio
import sys
import aiohttp
import discord
from discord.gateway import DiscordWebSocket, _log
from discord.ext.commands import Bot
class MyGateway(DiscordWebSocket):
async def identify(self):
payload = {
'op': self.IDENTIFY,
'd': {
'token': self.token,
'properties': {
'$os': sys.platform,
'$browser': 'Discord Android',
'$device': 'Discord Android',
'$referrer': '',
'$referring_domain': ''
},
'compress': True,
'large_threshold': 250,
'v': 3
}
}
if self.shard_id is not None and self.shard_count is not None:
payload['d']['shard'] = [self.shard_id, self.shard_count]
state = self._connection
if state._activity is not None or state._status is not None:
payload['d']['presence'] = {
'status': state._status,
'game': state._activity,
'since': 0,
'afk': False
}
if state._intents is not None:
payload['d']['intents'] = state._intents.value
await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
await self.send_as_json(payload)
_log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
class MyBot(Bot):
async def connect(self, *, reconnect: bool = True) -> None:
"""|coro|
Creates a websocket connection and lets the websocket listen
to messages from Discord. This is a loop that runs the entire
event system and miscellaneous aspects of the library. Control
is not resumed until the WebSocket connection is terminated.
Parameters
-----------
reconnect: :class:`bool`
If we should attempt reconnecting, either due to internet
failure or a specific failure on Discord's part. Certain
disconnects that lead to bad state will not be handled (such as
invalid sharding payloads or bad tokens).
Raises
-------
:exc:`.GatewayNotFound`
If the gateway to connect to Discord is not found. Usually if this
is thrown then there is a Discord API outage.
:exc:`.ConnectionClosed`
The websocket connection has been terminated.
"""
backoff = discord.client.ExponentialBackoff()
ws_params = {
'initial': True,
'shard_id': self.shard_id,
}
while not self.is_closed():
try:
coro = MyGateway.from_client(self, **ws_params)
self.ws = await asyncio.wait_for(coro, timeout=60.0)
ws_params['initial'] = False
while True:
await self.ws.poll_event()
except discord.client.ReconnectWebSocket as e:
_log.info('Got a request to %s the websocket.', e.op)
self.dispatch('disconnect')
ws_params.update(sequence=self.ws.sequence, resume=e.resume, session=self.ws.session_id)
continue
except (OSError,
discord.HTTPException,
discord.GatewayNotFound,
discord.ConnectionClosed,
aiohttp.ClientError,
asyncio.TimeoutError) as exc:
self.dispatch('disconnect')
if not reconnect:
await self.close()
if isinstance(exc, discord.ConnectionClosed) and exc.code == 1000:
# clean close, don't re-raise this
return
raise
if self.is_closed():
return
# If we get connection reset by peer then try to RESUME
if isinstance(exc, OSError) and exc.errno in (54, 10054):
ws_params.update(sequence=self.ws.sequence, initial=False, resume=True, session=self.ws.session_id)
continue
# We should only get this when an unhandled close code happens,
# such as a clean disconnect (1000) or a bad state (bad token, no sharding, etc)
# sometimes, discord sends us 1000 for unknown reasons so we should reconnect
# regardless and rely on is_closed instead
if isinstance(exc, discord.ConnectionClosed):
if exc.code == 4014:
raise discord.PrivilegedIntentsRequired(exc.shard_id) from None
if exc.code != 1000:
await self.close()
raise
retry = backoff.delay()
_log.exception("Attempting a reconnect in %.2fs", retry)
await asyncio.sleep(retry)
# Always try to RESUME the connection
# If the connection is not RESUME-able then the gateway will invalidate the session.
# This is apparently what the official Discord client does.
ws_params.update(sequence=self.ws.sequence, resume=True, session=self.ws.session_id)
bot = MyBot(command_prefix="?")
@bot.event
async def on_ready():
print(f"Sucessfully logged in as {bot.user}")
bot.run("YOUR_BOT_TOKEN")
Personally, I think that the following approach, which does include some runtime monkey-patching (but no AST manipulation) is cleaner for this purpose:
import sys
from discord.gateway import DiscordWebSocket, _log
from discord.ext.commands import Bot
async def identify(self):
payload = {
'op': self.IDENTIFY,
'd': {
'token': self.token,
'properties': {
'$os': sys.platform,
'$browser': 'Discord Android',
'$device': 'Discord Android',
'$referrer': '',
'$referring_domain': ''
},
'compress': True,
'large_threshold': 250,
'v': 3
}
}
if self.shard_id is not None and self.shard_count is not None:
payload['d']['shard'] = [self.shard_id, self.shard_count]
state = self._connection
if state._activity is not None or state._status is not None:
payload['d']['presence'] = {
'status': state._status,
'game': state._activity,
'since': 0,
'afk': False
}
if state._intents is not None:
payload['d']['intents'] = state._intents.value
await self.call_hooks('before_identify', self.shard_id, initial=self._initial_identify)
await self.send_as_json(payload)
_log.info('Shard ID %s has sent the IDENTIFY payload.', self.shard_id)
DiscordWebSocket.identify = identify
bot = Bot(command_prefix="?")
@bot.event
async def on_ready():
print(f"Sucessfully logged in as {bot.user}")
bot.run("YOUR_DISCORD_TOKEN")
As to why editing the library source code did not work for you, I can only assume that you have edited the wrong copy of the file, as people have commented.
QUESTION
SOLVED - The categories/channels were set to private which is why they couldn't be changed even with the correct permissions. My solution was to tell the user which ones and to add the role with the permissions manually. I added a message that said they can just give the bot administrator permissions and it will do it for them. Hopefully, this helps anyone who runs into this in the future.
I'm trying to change the permissions of multiple categories and channels using the following code:
role = get(guild.roles, name="Fun")
for channel in guild.channels:
overwrites = channel.overwrites_for(role)
overwrites.view_channel = False
await channel.set_permissions(role, overwrite=overwrites)
I get the following error:
await channel.set_permissions(role, overwrite=overwrites)
discord.errors.Forbidden: 403 Forbidden (error code: 50013): Missing Permissions
It will work if I give the bot the administrator permission. However, it won't work if I give the bot every single other permission, the bot is at the top of the roles, and it has every category permission. What am I missing? I'm using Novus (fork of discord.py so it's pretty much the same thing).
Edit: I've asked people in the discord.py, Novus, and discord developers discord servers and they can't help me either. With the exception of the discord developers server they just didn't answer my question.
Edit 2: I was able to get it working when I manually added the bot's role to the category and enabled view_channel. This won't work, however, because that means the server owner has to manually do this for every category and channel which is really inconvenient.
ANSWER
Answered 2021-Oct-31 at 12:05This is simply how Discord works, if a channel is private and the bot does not have administrator permissions or role that has permissions to view the channel, it is not able to view it.
You can create a new category that the bot can access like this:
role = await guild.create_role(name="Fun")
category = await guild.create_category("category_name")
await category.set_permissions(role, read_messages=True, send_messages=True, connect=True, speak=True)
await category.set_permissions(ctx.guild.me, read_messages=True, send_messages=True, speak=True)
QUESTION
I'm trying to make a Discord bot that just says if someone is online on the game.
However I keep getting this message:
[ERR_REQUIRE_ESM]: require() of ES Module from not supported. Instead change the require of index.js in... to a dynamic import() which is available in all CommonJS modules.
This is my code:
module.exports = {
name: 'username',
description: "this is the username command",
async execute(message, args) {
const fetch = require('node-fetch');
if (args.length !== 1) {
return message.channel.send("invalid username wtf")
}
const ign = args[0]
if (ign.length > 16 || ign.length < 3) {
return message.channel.send("invalid username wtf")
}
const uuid = await fetch(`https://api.mojang.com/users/profiles/minecraft/${ign}`).then(data => data.json()).then(data => data.id).catch(err => message.channel.send("error wtf"));
const onlineInfo = await fetch(`https://api.hypixel.net/status?key=${john}&uuid=${uuid}`).then(data => data.json());
if (uuid.length !== 32) {
return;
}
if (onlineinfo.success) {
if (onlineinfo.session.online) {
message.channel.send("they are online")
}
else {
message.channel.send("they are offline")
}
}
else {
message.channel.send("hypixel api bad wtf")
}
}
}
This is my package.json file:
{
"name": "discordbot",
"version": "1.0.0",
"main": "main.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "node main.js"
},
"author": "",
"license": "ISC",
"description": "",
"dependencies": {
"discord.js": "^13.0.1",
"node-fetch": "^3.0.0"
}
}
ANSWER
Answered 2021-Sep-07 at 06:38node-fetch v3 recently stopped support for the require way of importing it in favor of ES Modules. You'll need to use ESM imports now, like:
import fetch from "node-fetch";
at the top of your file.
QUESTION
If a user has the privacy setting "Allow direct messages from server members" turned off and a discord bot calls
await user.dm_channel.send("Hello there")
You'll get this error:
discord.errors.Forbidden: 403 Forbidden (error code: 50007): Cannot send messages to this user
I would like to check whether I can message a user without sending them a message. Trying to send a message and catching this error does not work for me, because I don't want a message to get sent in the event that the bot is allowed to message.
I have tried this:
print(user.dm_channel.permissions_for(bot).send_messages)
but it always returns True, even if the message is not permitted.
I have also tried this:
channel = await user.create_dm()
if channel is None:
...
but unfortunately, it seems that "has permission to message user" and "has permission to create a dm channel" are considered different.
EDIT
To clarify the exact usage since there seems to be a bit of confusion, take this example. There is a server, and 3 users in question: Me, My Bot, and Steve. Steve has "Allow direct messages from server members" checked off.
The bot has a command called !newgame which accepts a list of users and starts a game amongst them, which involves DMing some of the members of the game. Because of Steve's privacy settings, he cannot play the game (since the bot will need to message him). If I do
!newgame @DJMcMayhem @Steve
I'd like to provide a response like:
> I can't start a game with that list of users because @Steve has the wrong privacy settings.
But as far as I know right now, the only way to find out if Steve can play is by first attempting to message every user, which I'd like to avoid.
ANSWER
Answered 2022-Jan-22 at 22:03You can generate a Bad Request to the dm_channel. This can be accomplished by setting content to None, for example.
If it returns with 400 Bad Request, you can DM them. If it returns with 403 Forbidden, you can't.
async def can_dm_user(user: discord.User) -> bool:
ch = user.dm_channel
if ch is None:
ch = await user.create_dm()
try:
await ch.send()
except discord.Forbidden:
return False
except discord.HTTPException:
return True
QUESTION
I am working on certain stock-related projects where I have had a task to scrape all data on a daily basis for the last 5 years. i.e from 2016 to date. I particularly thought of using selenium because I can use crawler and bot to scrape the data based on the date. So I used the use of button click with selenium and now I want the same data that is displayed by the selenium browser to be fed by scrappy. This is the website I am working on right now. I have written the following code inside scrappy spider.
class FloorSheetSpider(scrapy.Spider):
name = "nepse"
def start_requests(self):
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
floorsheet_dates = ['01/03/2016','01/04/2016', up to till date '01/10/2022']
for date in floorsheet_dates:
driver.get(
"https://merolagani.com/Floorsheet.aspx")
driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
).send_keys(date)
driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
total_length = driver.find_element(By.XPATH,
"//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
z = int((total_length.split()[-1]).replace(']', ''))
for data in range(z, z + 1):
driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
self.url = driver.page_source
yield Request(url=self.url, callback=self.parse)
def parse(self, response, **kwargs):
for value in response.xpath('//tbody/tr'):
print(value.css('td::text').extract()[1])
print("ok"*200)
Update: Error after answer is
2022-01-14 14:11:36 [twisted] CRITICAL:
Traceback (most recent call last):
File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/home/navaraj/PycharmProjects/first_scrapy/env/lib/python3.8/site-packages/scrapy/crawler.py", line 88, in crawl
start_requests = iter(self.spider.start_requests())
TypeError: 'NoneType' object is not iterable
I want to send current web html content to scrapy feeder but I am getting unusal error for past 2 days any help or suggestions will be very much appreciated.
ANSWER
Answered 2022-Jan-14 at 09:30The 2 solutions are not very different. Solution #2 fits better to your question, but choose whatever you prefer.
Solution 1 - create a response with the html's body from the driver and scraping it right away (you can also pass it as an argument to a function):
import scrapy
from selenium import webdriver
from selenium.webdriver.common.by import By
from scrapy.http import HtmlResponse
class FloorSheetSpider(scrapy.Spider):
name = "nepse"
def start_requests(self):
# driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver = webdriver.Chrome()
floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
for date in floorsheet_dates:
driver.get(
"https://merolagani.com/Floorsheet.aspx")
driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
).send_keys(date)
driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
total_length = driver.find_element(By.XPATH,
"//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
z = int((total_length.split()[-1]).replace(']', ''))
for data in range(1, z + 1):
driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
self.body = driver.page_source
response = HtmlResponse(url=driver.current_url, body=self.body, encoding='utf-8')
for value in response.xpath('//tbody/tr'):
print(value.css('td::text').extract()[1])
print("ok"*200)
# return an empty requests list
return []
Solution 2 - with super simple downloader middleware:
(You might have a delay here in parse method so be patient).
import scrapy
from scrapy import Request
from scrapy.http import HtmlResponse
from selenium import webdriver
from selenium.webdriver.common.by import By
class SeleniumMiddleware(object):
def process_request(self, request, spider):
url = spider.driver.current_url
body = spider.driver.page_source
return HtmlResponse(url=url, body=body, encoding='utf-8', request=request)
class FloorSheetSpider(scrapy.Spider):
name = "nepse"
custom_settings = {
'DOWNLOADER_MIDDLEWARES': {
'tempbuffer.spiders.yetanotherspider.SeleniumMiddleware': 543,
# 'projects_name.path.to.your.pipeline': 543
}
}
driver = webdriver.Chrome()
def start_requests(self):
# driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
floorsheet_dates = ['01/03/2016','01/04/2016']#, up to till date '01/10/2022']
for date in floorsheet_dates:
self.driver.get(
"https://merolagani.com/Floorsheet.aspx")
self.driver.find_element(By.XPATH, "//input[@name='ctl00$ContentPlaceHolder1$txtFloorsheetDateFilter']"
).send_keys(date)
self.driver.find_element(By.XPATH, "(//a[@title='Search'])[3]").click()
total_length = self.driver.find_element(By.XPATH,
"//span[@id='ctl00_ContentPlaceHolder1_PagerControl2_litRecords']").text
z = int((total_length.split()[-1]).replace(']', ''))
for data in range(1, z + 1):
self.driver.find_element(By.XPATH, "(//a[@title='Page {}'])[2]".format(data)).click()
self.body = self.driver.page_source
self.url = self.driver.current_url
yield Request(url=self.url, callback=self.parse, dont_filter=True)
def parse(self, response, **kwargs):
print('test ok')
for value in response.xpath('//tbody/tr'):
print(value.css('td::text').extract()[1])
print("ok"*200)
Notice that I've used chrome so change it back to firefox like in your original code.
QUESTION
Scrapping links should be a simple feat, usually just grabbing the src value of the a tag.
I recently came across this website (https://sunteccity.com.sg/promotions) where the href value of a tags of each item cannot be found, but the redirection still works. I'm trying to figure out a way to grab the items and their corresponding links. My typical python selenium code looks something as such
all_items = bot.find_elements_by_class_name('thumb-img')
for promo in all_items:
a = promo.find_elements_by_tag_name("a")
print("a[0]: ", a[0].get_attribute("href"))
However, I can't seem to retrieve any href, onclick attributes, and I'm wondering if this is even possible. I noticed that I couldn't do a right-click, open link in new tab as well.
Are there any ways around getting the links of all these items?
Edit: Are there any ways to retrieve all the links of the items on the pages?
i.e.
https://sunteccity.com.sg/promotions/724
https://sunteccity.com.sg/promotions/731
https://sunteccity.com.sg/promotions/751
https://sunteccity.com.sg/promotions/752
https://sunteccity.com.sg/promotions/754
https://sunteccity.com.sg/promotions/280
...
ANSWER
Answered 2022-Jan-15 at 19:47You are using a wrong locator. It brings you a lot of irrelevant elements.
Instead of find_elements_by_class_name('thumb-img') please try find_elements_by_css_selector('.collections-page .thumb-img') so your code will be
all_items = bot.find_elements_by_css_selector('.collections-page .thumb-img')
for promo in all_items:
a = promo.find_elements_by_tag_name("a")
print("a[0]: ", a[0].get_attribute("href"))
You can also get the desired links directly by .collections-page .thumb-img a locator so that your code could be:
links = bot.find_elements_by_css_selector('.collections-page .thumb-img a')
for link in links:
print(link.get_attribute("href"))
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install InstaPy
You can use InstaPy like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesExplore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits
Save this library and start creating your kit
Share this Page
