PyTables | A Python package to manage extremely large amounts of data | Data Visualization library

What works:

1. make a new compute instance
2. open it up using JupyterLab
3. open a new terminal
4. conda activate azureml_py36
5. conda install seaborn -y
6. open a new notebook and run import seaborn as sns

1. Are you using the kernel, Python 3.6 - AzureML (i.e. the azureml_py36 conda env)?
2. Have you tried restarting the kernel and/or creating a new compute instance?

True is 0000 0001. Hence ~True yields 1111 1110, which is -2 in two's-complement representation.

False is 0000 0000. Hence ~False yields 1111 1111, which is -1.

I follow the instructions listed here: https://rasterio.readthedocs.io/en/latest/installation.html
This page also has separate instructions for those using Anaconda.

The GDAL installation is by far the most annoying but once it's done, the hard part is over. The python utilities for both rasterio and gdal can be found here:
https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal
The second link is also provided on the PyPi page but I like to keep it bookmarked because there's a lot of good resources there!

Without seeing your data types, it's hard to give specific advice.
First question: do you really want to create a Table or will an Array suffice? PyTables has both types. What's the difference?
An Array holds homogeneous data (like a NumPy ndarray) and can have any dimension. An Table is typically used to hold heterogeneous data (like a NumPy recarray) and is always 2d (really a 1d array of structured types). Tables also support complex queries with the PyTables API.

The key when creating a Table is to either use the description= or obj= parameter to describe the structured types (and field names) for each row. I recently posted an answer that shows how to create a Table. Please review. You may find you don't want to create 200000 fields/columns to define the Table. See this answer: different data types for different columns of an array

If you just want to save a matrix of 200000x200000 homogeneous entities, an array is easier. (Given the data size, you probably need to use an EArray, so you can write the data in increments.) I wrote a simple example that creates an EArray with 2000x200000 entities, then adds 3 more sets of data (each 2000 rows; total of 8000 rows).

• The shape=(0,nrows) parameter indicates the first axis can be extended, and creates ncols columns.
• The expectedrows=nrows parameter is important in large datasets to improvie I/O performance.

The resulting HDF5 file is 6GB. Repeat earr.append(arr) 99 times to get 200000 rows. Code below: