TwitterBot | Golang smart Twitter bot | Bot library

Lately @MrWhite gave us another, better and simple solution - just add DirectoryIndex index.html to .htaccess file will do the same.

From the beginning I wrote that DirectoryIndex is working but NO! It seems it's working when you try prerender.io, but in reality it was showing website like this:

and I had to remove it. So it was not issue with .htaccess file, it was coming from the server.

What I did was I went into WHM->Apache Configurations->DirectoryIndex Priority and I saw this list

and yes that was it!

To fix I just moved index.html to the very top second comes index.html.var and after rest of them.

I don't know what index.html.var is for, but I did not risk just to remove it. Hope it helps someone who struggled as me.

Thanks to @CBroe's guidance, I managed to make the social media (Facebook and Twitter) crawlers work (without using Angular Universal, Scully, Prerender.io, etc) for an Angular 11 SPA - Single Page Application, which I'm hosting on a shared hosting server.

The issue I had in the question above was in .htaccess.

This is my .htaccess (which works as expected):

I took a while to figure it out. The CLI command allows to pass --content-type. In the documentation is the example application/json. For regular data, it would be form data and urlencoded. The following works, i.e., using application/x-www-form-urlencoded:

Could you please try following, written with your shown samples. Please make sure to clear your browser cache before testing your URLs.

I think you can instead use the Twitter API with the requests library (It's pip install requests I think).

Here's the link to the documentation: https://developer.twitter.com/en/docs/twitter-api

The error is that on the line

self.browser = webdriver.Chrome("C:\Users\sfdxs\Downloads\chromedriver")

you forgot to add .exe so the line should be

self.browser = webdriver.Chrome("C:\Users\sfdxs\Downloads\chromedriver.exe")

instead.

Also, Twitter probably has a reCAPTCHA(something that prevents bots) on the login page so I suggest that you use their API.

You have small typo in your xpath.

Checking the Twitter Card Validator right now shows the correct image. Sometimes the card you're seeing is cached, so if you want to look at the latest version of the image, just add a parameter with a random name and (optionally) a random value.

For the link https://speakher.jp, for example, just adding a small random parameter will force Twitter to load the latest version of the image. This should load the same page since the parameter means nothing (normally).

So my error was assuming that the enviroment is portable, which it isn't, and does not make much sense in the first place. At creation the enviroment saves the path to the python exe, so when that path is different on another workstation, of course it does not work.

Like @Panagiotis Kanavos answered: I should have a different enviroment, preferably with the same name, on every workstation. These enviroments should not be included in version control. This works as it should.

I think that Selenium only works with web page. You could try with requests but it's very difficult to scrape/work with Twitter without Twitter API..