di_textanalysis | Web scraping and text analysis using SAP Data Intelligence

di_textanalysis is a Python library. di_textanalysis has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. However di_textanalysis build file is not available. You can download it from GitHub.

There are basically 2 approaches for doing a text analysis:. The brute force approach is using a text as a whole and classifies the text by using 'Deep Learning' techniques. The most familiar classifications are the sentiment and subjectivity polarity. Based on the type of the corpus the result could be quite good even for short texts like in twitter or movie and product reviews. But due to the very nature of language of being able to convey all kinds of information in numerous ways and because you need mostly an extensive amount of trained data this approach is of limited practicality. The second step-by-step approach could also use models trained by deep learning techniques but in a more controlled way and combined with other techniques in a kind of process pipeline. For example for the grammatical analysis a deep learning trained model could be of great use to create word bags that can subsequently been evaluated to find similar topics in texts by applying cluster algorithms. For a proof-of-concept we have done both ways. A simple sentiment scoring of the texts and a word indexing pipeline. The word index can then be used for further researches like the. As a text corpus we are using online media articles that we scrape on a daily basis. For a start we selected 2 French (Le Figaro, Le Monde), 2 Spanish (El Mundo, El Pais) and 3 German newspaper (Der Spiegel, FAZ, Süddeutsche).