wiki_processor | process enwiki-latest-pages-articles.xml , and parse

wiki_processor is a Python library. wiki_processor has no bugs, it has no vulnerabilities and it has low support. However wiki_processor build file is not available. You can download it from GitHub.

#wiki processor ##additional scripts run with the following order: compute_hits.py, compute_pagerank.py. run all the python scripts from successornodes.py to other_topic_feature.py in batch. this will cost 47min7sec on 51 topics on server. the final result is stored in data/final_words/.*_final_words.txt. ####compute_hits.py compute hits score of categories according to the edges from categories to pages. and this hits score are used as prior knowledge in page rank computing. ####compute_pagerank.py compute page rank score of categories according to the edges from categories to their subcategories. the hits score are used in personalized page rank algorithm. input: cattopagegraph.txt (category->sub category) and hub_score.txt (used as the prior knowledge in personalized page rank algorithm). ####rebuildtaxonomy.py remove the edges that smaller pagerank-score nodes pointing to higer pagerank-score nodes. this step can make sure that the taxonomy graph is the directed acyclic graph. usage : python rebuildtaxonomy.py. input: pagerank_score.txt and taxonomygraph.txt. ####successornodes.py find out the successor nodes in a directed acyclic graph (dag) given a node. usage: python successornodes.py [node="military"]. ####pages.py find out the wiki article pages related to a specific topic such as military, baseball, film and so on. and this will cost 6m28.876s and 20 gb memory on server. usage: python pages.py [node="military" | node ="all"] node="all" means to process all topics mentioned in