linguist | Easy multilingual urls and redirection support | Internationalization library

1. Change your html structure

Nothing guys, the text in the properties file just had to be edited to:

getstarted=commen\u1E09\ons

Yes, the default tokenizer is OpenNLP and it works with no configuration needed. It will guess the language if you don't set it, but if you know the document language it is better to use set_language (and language=...) in queries, since language detection is unreliable on short text.

However, OpenNLP tokenization (not detecting) only supports Danish, Dutch, Finnish, French, German, Hungarian, Irish, Italian, Norwegian, Portugese, Romanian, Russian, Spanish, Swedish, Turkish and English (where we use kstem instead). So, no CJK.

To support CJK you need to plug in your own tokenizer as described in the linguistics doc, or else use ngram instead of tokenization, see https://docs.vespa.ai/documentation/reference/schema-reference.html#gram

n-gram is often a good choice with Vespa because it doesn't suffer from the recall problems of CJK tokenization, and by using a ranking model which incorporates proximity (such as e.g nativeRank) you'l still get good relevancy.

That's a very small dataset (100 docs) and vocabulary (100 words) compared to much published work of Doc2Vec, which has usually used tens-of-thousands or millions of distinct documents.

That each doc is thousands of words and you're using PV-DM mode that mixes both doc-to-word and word-to-word contexts for training helps a bit. I'd still expect you might need to use a smaller-than-defualt dimensionaity (vector_size<<100), & more training epochs - but if it does seem to be working for you, great.

You don't mention how many classes you have, nor what classifier algorithm you're using, nor whether known classes are being mixed into the (often unsupervised) Doc2Vec training mode.

If you're only using known classes as the doc-tags, and your "a few" classes is, say, only 3, then to some extent you only have 3 unique "documents", which you're training on in fragments. Using only "a few" unique doctags might be prematurely hiding variety on the data that could be useful to a downstream classifier.

On the other hand, if you're giving each doc a unique ID - the original 'Paragraph Vectors' paper approach, and then you're feeding those to a downstream classifier, that can be OK alone, but may also benefit from adding the known-classes as extra tags, in addition to the per-doc IDs. (And perhaps if you have many classes, those may be OK as the only doc-tags. It can be worth comparing each approach.)

I haven't seen specific work on making Doc2Vec models explainable, other than the observation that when you are using a mode which co-trains both doc- and word- vectors, the doc-vectors & word-vectors have the same sort of useful similarities/neighborhoods/orientations as word-vectors alone tend to have.

You could simply try creating synthetic documents, or tampering with real documents' words via targeted removal/addition of candidate words, or blended mixes of documents with strong/correct classifier predictions, to see how much that changes either (a) their doc-vector, & the nearest other doc-vectors or class-vectors; or (b) the predictions/relative-confidences of any downstream classifier.

(A wishlist feature for Doc2Vec for a while has been to synthesize a pseudo-document from a doc-vector. See this issue for details, including a link to one partial implementation. While the mere ranked list of such words would be nonsense in natural language, it might give doc-vectors a certain "vividness".)

Whn you're not using real natural language, some useful things to keep in mind:

• if your 'texts' are really unordered bags-of-tokens, then window may not really be an interesting parameter. Setting it to a very-large number can make sense (to essentially put all words in each others' windows), but may not be practical/appropriate given your large docs. Or, trying PV-DBOW instead - potentially even mixing known-classes & word-tokens in either tags or words.
• the default ns_exponent=0.75 is inherited from word2vec & natural-language corpora, & at least one research paper (linked from the class documentation) suggests that for other applications, especially recommender systems, very different values may help.

you are trying to add to MySQL bs4 tag:

but it didn't fix the issue

Yes. What you need to do is, asking GitHub Linguist for NOT seeing your CSS files (that is what you did now), or regarding your CSS files as some other types for example JavaScript.

Is tailwind css creating some hidden files which is shooting up CSS usage ?

Yes. While opening your repository, I found a 3.4MB "large" file src/styles/app.css which is probably the trouble maker.

Your comment said, "add to .gitignore and check". Please think twice:

1. Is your code run without this file? (or another question, is this file generated during code deploying which means your code can still deploys without this file?)
2. It is not enough for you to just add that file in .gitignore: Without other operation such as removing it from your Git repository (either via deleting it then git add, or use git filter-branch or bfg to remove the file from entire Git repository), it will be "useless" since that this file has already tracked by Git.

I believe the cleanest way to handle this is to specifically narrow the type of category to keyof NormativeCaseType. What that does is restrict the values to strings that are the property names.

As indicated by file, your program is a Linux application so obviously you can't run it on Windows. See

• Why does a linux compiled program not work on Windows
• Why won't Windows EXE files work on Linux?
• Why do you need to recompile C/C++ for each OS?

Mingw is not an environment for running Linux executables, it's just a compiler that compiles POSIX code into native Windows binaries. Neither is Cygwin, which is a reimplementation of POSIX system calls in Windows, and Cygwin binaries are also native Windows binaries with a dependency on Cygwin DLLs. Read this if you want to know their differences. Bash is a shell and isn't a platform to execute files either. Only the runtime platform (OS or something like JVM or .NET CLR VM) can run programs and a shell is just a tool to interact with the OS

So you must run Linux programs in a Linux environment like a Linux machine or WSL1/2. Since the program is 32-bit, you can only run it in Linux or WSL2

Since you have the source code you can also compile the code with mingw or cygwin and run on Windows

I am wondering if you have you seen PyTextRank or spacycaKE extension to SpaCy?

Both can help with phrase extraction which is not possible directly with SpaCy.