gazetteer | OSM ElasticSearch geocoder and addresses | Map library

I have the following Models/Tables in falsk-sqlalchemy.:

I have the following tables (simplified version):

I have followed this SpaCy tutorial for training a custom dataset. My dataset is a gazetteer. Therefore, I made my training data as the following.

Switching to the new release of Gate (from the 8.4.1 to the 8.5.1), it seems not possible to modify the ANNIE gazetteer adding a new list. In fact in the gazetter editor the box for adding a new list it's disabled. I've tried also to look for the files usually located in C:\Program Files\GATE_Developer_8.4.1\plugins\ANNIE\resources\gazetteer, but the plugin folder is not there.

I want to know how common certain place names are. From a national gazetteer, I did two smoothScatter() in R, one with all the places, other with the places whose names I'm interested in.

All places:

Places with certain names:

Now, how can I divide the second by the first, to get the density of the names of interest over all names? It can be a R solution, or ImageMagick, GIMP...

I am in the midst of re-writing a very old VB legacy application into a browser based C# ASP.Net application using Core 2.1 and Entity Framework back-ended to SQL server.

Several of the functions are long-running tasks. One example is an import of an address gazetteer CSV file. The files are typically 50-100Mb in size and need parsing. I have written an uploader and import function, which runs in around 15 minutes - most of that is database write-time.

I am trying to find a way to run the import process so that it can report back progress to the client browser, ideally by changing the menu option to a progress bar until the task is done - since the _layout.cshtml is on every page, it would let any user know the task is running and when it will finish.

I've looked at IHostedServices and the BackgroundService functions, but I cannot find any examples that match what I'm trying to do. I've seen an article around MVC5 that used SignalR & Knockout (which I'm less familiar with) but it doesn't use the Core 2+ or the newer service functions.

Can anyone point me to a good .Net Core > 2.0 example of something like this?

Thanks in advance.

I'm trying to work with JSON file on spark (pyspark) environment.

Problem: Unable to convert JSON to expected format in Pyspark Dataframe

1st Input data set:

https://health.data.ny.gov/api/views/cnih-y5dw/rows.json

In this file metadata is defined at start for of the file with tag "meta" and then followed by data with tag "data".

FYI: Steps taken to download data from web to local drive. 1. I've downloaded file to my local drive 2. then pushed to hdfs - from there I'm reading it to spark environment.

First, if you haven't seen the Dedupe library for Python: it's awesome. Much like TensorFlow, it's a great way to bring machine learning to the masses (like me).

I'm trying to do record linkage of names against a single, large, messy data set. I'm using heuristics right now, and it's starting to fall short with more complicated data sets.

Questions:

Is there a way to perform a match of a single record (one-by-one or in batches) and return all the potential matches?

Gazetteer docs say one side must be clean, no duplicates. If names can be duplicated but serial numbers aren't (and serial numbers aren't used in matching) isn't that a duplicate?

Context:

There are 1.6M specialized construction machines in the US. There is a database with the machine type, owner names (up to two, companies included), serial number, and maintenance information like last_service_date.

People often inquire about maintenance and sales of their machines (100-250/day), and I keep a running record. The problem is matching the name on the phone with the machine(s) that they own. I need to match the names I have on the forms with the names on the ownership records to learn more about the machine after the fact and understand the lifecycle of the machines.

Sample Data:

Evaluating Dataflow, and am trying to figure out if/how to do the following.

My apologies if anything in the above is trivial--trying to wrap our heads around Dataflow before we make a decision on using Beam, or something else like Spark, etc.

General use case is for machine learning:

• Ingesting documents which are individually processed.

• In addition to easy-to-write transforms, we'd like to enrich each document based on queries against databases (that are largely key-value stores).

• A simple example would be a gazetteer: decompose the text into ngrams, and then check if those ngrams reside in some database, and record (within a transformed version of the original doc) the entity identifier given phrases map to.

How to do this efficiently?

NAIVE (although possibly tricky with the serialization requirement?):

Each document could simply query the database individually (similar to Querying a relational database through Google DataFlow Transformer), but, given that most of these are simple key-value stores, it seems like there should be a more efficient way to do this (given the real problems with database query latency).

SCENARIO #1: Improved?:

Current strawman is to store the tables in Bigquery, pull them down (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py), and then use them as side inputs, that are used as key-value lookups within the per-doc function(s).

Key-value tables range from generally very small to not-huge (100s of MBs, maybe low GBs). Multiple CoGroupByKey with same key apache beam ("Side inputs can be arbitrarily large - there is no limit; we have seen pipelines successfully run using side inputs of 1+TB in size") suggests this is reasonable, at least from a size POV.

1) Does this make sense? Is this the "correct" design pattern for this scenario?

2) If this is a good design pattern...how do I actually implement this?

https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L53 shows feeding the result to the document function as an AsList.

i) Presumably, AsDict is more appropriate here, for the above use case? So I'd probably need to run some transformations first on the Bigquery output to separate it into key, value tuple; and make sure that the keys are unique; and then use it as a side input.

ii) Then I need to use the side input in the function.

What I'm not clear on:

• for both of these, how to manipulate the output coming off of the Bigquery pull is murky to me. How would I accomplish (i) (assuming it is necessary)? Meaning, what does the data format look like (raw bytes? strings? is there a good example I can look into?)

• Similarly, if AsDict is the correct way to pass it into the func, can I just reference things like a dict normally is used in python? e.g., side_input.get('blah') ?

SCENARIO #2: Even more improved? (for specific cases):

• The above scenario--if achievable--definitely does seem like it is superior continuous remote calls (given the simple key-value lookup), and would be very helpful for some of our scenarios. But if I take a scenario like a gazetteer lookup (like above)...is there an even more optimized solution?

Something like, for every doc, writing our all the ngrams as keys, with values as the underlying indices (docid+indices within the doc), and then doing some sort of join between these ngrams and the phrases in our gazeteer...and then doing another set of transforms to recover the original docs (now w/ their new annotations).

I.e., let Beam handle all of the joins/lookups directly?

Theoretical advantage is that Beam may be a lot quicker in doing this than, for each doc, looping over all of the ngrams and doing a check if the ngram is in the side_input.

Other key issues:

3) If this is a good way to do things, is there any trick to making this work well in the streaming scenario? Text elsewhere suggests that the side input caching works more poorly outside the batch scenario. Right now, we're focused on batch, but streaming will become relevant in serving live predictions.

4) Any Beam-related reason to prefer Java>Python for any of the above? We've got a good amount of existing Python code to move to Dataflow, so would heavily prefer Python...but not sure if there are any hidden issues with Python in the above (e.g., I've noticed Python doesn't support certain features or I/O).

EDIT: Strawman? for the example ngram lookup scenario (should generalize strongly to general K:V lookup)

• Phrases = get from bigquery
• Docs (indexed by docid) (direct input from text or protobufs, e.g.)
• Transform: phrases -> (phrase, entity) tuples
• Transform: docs -> ngrams (phrase, docid, coordinates [in document])
• CoGroupByKey key=phrase: (phrase, entity, docid, coords)
• CoGroupByKey key=docid, group((phrase, entity, docid, coords), Docs)
• Then we can iteratively finalize each doc, using the set of (phrase, entity, docid, coords) and each Doc