ijson | efficient alternative to serde_json : : Value | JSON Processing library

The following program assumes that the input files have a format that is predictable enough to skip JSON parsing for the sake of performance.

My assumptions, inferred from your description, are:

• All files have the same encoding.
• All files have a single position somewhere at the start where "array_element":[ can be found, after which the "interesting portion" of the file begins
• All files have a single position somewhere at the end where ]} marks the end of the "interesting portion"
• All "interesting portions" can be joined with commas and still be valid JSON

When all of these points are true, concatenating a predefined header fragment, the respective file ranges, and a footer fragment would produce one large, valid JSON file.

Beware of how you are collecting the results from kvitems. In all your examples above you are using a generator expression, which are themselves lazy-evaluated, and this may lead to misunderstandings. You are not showing however how you find that your final dictionary has values for id but not for the other keys. I'm assuming it's only because you are iterating over the values under the parse_records['id'] values first. As you do so, the generator expression is then evaluated and the underlying kvitems generator is exhausted. When you iterate over the values of the other generator expressions, the underlying kvitems generator that feeds them is exhausted so they yield nothing. However, if you were to iterate over the values for one of the other keys first, you should see values for that key and not for the others.

Generator expressions themselves are great, but in this case it might end up adding confusion. If you want to avoid this situation you may want to consolidate those sequences to be lists instead (e.g., using [... for k, v in kvitems ...] instead of (... for k, v in kvitems ...)).

As you point out, kvitems is a single-pass generator (or a single-pass asynchronous generator when fed with an asynchronous file-like object), so once you fully iterate over it, further iterations yield no values. This is why indeed in your original code you get values for id but not for the other keys that are collected on subsequent iterations over an already-iterated kvitems object.

Trying to duplicate the kvitems object is also bogus: as you also found out, you are simply creating a list with the same object in all positions instead of copies of the original object.

Trying to copy the kvitems is simply not possible. The only option to get a N "copies" is to actually construct N different object; this means however that the input file will be read N times (and needs to be opened N times as well, as kvitems will advance the given file until it doesn't have any more input). Possible, but not great.

The result of itertools.cycle is an infinite generator. Then you use this as the basis to construct different generator expressions (so, lazy evaluated). You mention that this solution worked in ways "you don't understand", but don't delve on what exactly happened. My expectation is that when trying to inspect the values for any of the keys, you run into an infinite loop because your generator expression is iterating over an infinite generator, or something similar.

You say that your finally piece of code works as expected. This is the only bit that surprises me, specially if you really, really inspected (i.e., evaluated) all three of the generator expressions after you created them. If you could clarify if that's the case it would be interesting; otherwise if you created all three generator expressions, but then evaluated one or the other, then there are no surprises here (because of the "About result collection" explanation).

It basically all boils down to doing a single iteration over kvitems. You could try for instance something like this:

To directly answer your two questions:

• The decompression method is correct in the sense that it yields JSON data that you then feed to ijson. As you point out, ijson works both with str and bytes inputs (although the latter is preferred); if you were giving ijson some non-JSON input you wouldn't see an error showing JSON data in it.

• This is a very common error that is described in ijson's FAQ. It basically means your JSON document has more than one top-level value, which is not standard JSON, but is supported by ijson by using the multiple_values option (see docs for details).

About the code as a whole: while it's working correctly, it could be improved on: the whole point of using ijson is that you can avoid loading the full JSON contents in memory. The code you posted doesn't use this to its advantage though: it first opens the bz-compressed file, reads it as a whole, decompresses that as a whole, (unnecessarily) decodes that as a whole, and then gives the decoded data as input to ijson. If your input file is small, and the decompressed data is also small you won't see any impact, but if your files are big then you'll definitely start noticing it.

A better approach is to stream the data through all the operations so that everything happens incrementally: decompression, no decoding and JSON parsing. Something along the lines of:

This answer works if you are sure that the data is a GeoJSON and it is structured properly:

For reading GeoJSON data you can use Geopandas library:

I've had to make quite some extrapolations and assumptions, but it looks like

• you're using Django
• you want to populate an SQL database with venue, paper and author data
• you want to then do some analysis using Pandas

Populating your SQL database can be done pretty neatly with something like the following.

• I added the tqdm package so you get a progress indication.
• This assumes there's a PaperAuthor model that links papers and authors.
• Unlike the original code, this will not save duplicate Venues in the database.
• You can see I replaced get_or_create and create with stubs to make this runnable without the database models (or indeed, without Django), just having the dataset you're using available.

On my machine, this consumes practically no memory, as the records are (or would be) dumped into the SQL database, not into an ever-growing, fragmenting dataframe in memory.

The Pandas processing is left as an exercise for the reader ;-), but I'd imagine it'd involve pd.read_sql() to read this preprocessed data from the database.

You need to use ijson's event interception mechanism. Basically go one level down in the parsing logic by using ijson.parse until you hit the big array, then switch to using ijson.items with the rest of the parse events. This uses a string literal, but should illustrate the point:

I think if you need to keep track of CVE IDs and their corresponding CPEs you'll need to iterate over whole cve items and extract the bits of data you need (so you'll only do one pass through the file). Not as efficient memory-wise as your original iteration, but if each item in CVE_Items is not too big then it's not a problem:

have you tried this json.dump(row, temp, indent=4)

Your proposal is:

• Step 1 read json file
• Step 2 load to dataframe
• Step 3 save file as a csv
• Step 4 load csv to sql
• Step 5 load data to django to search

The problem with your second example is that you still use global lists (data_phone, data_name), which grow over time.

Here's what you should try, for huge files:

• Step 1 read json
  • line by line
  • do not save any data into a global list
  • write data directly into SQL
• Step 2 Add indexes to your database
• Step 3 use SQL from django

You don't need to write anything to CSV. If you really want to, you could simply write the file line by line: