columnar | High-throughput columnar serialization in Rust

I have a scenario where in my directory their are 10 files

each file as one columnar record like below

file1 :

I often take the columnar output of some command and pipe it to awk and xargs to perform some action that I want. A good example of this would be taking the output of docker ps, fetching the container IDs, and removing those containers. I understand that there are easier ways to do this with Docker, but in my case I want to post-process the list so I'm doing it the hard way. Anyway, the command looks something like docker ps -f status=exited | tail -n +2 | awk '{ print $1 }' | xargs docker rm. If I run this command directly, it works ok if there are containers that match. If the list is empty, however, awk still tries to pipe an empty string to xargs. This results in an error from docker rm that looks like this...

I'm new to python an I want to improve several excel programs I've made using VBA. Like the one below. I have a machine log which is consist of 2 Columns and average of 50,000 Rows, every group is separated by spaces. Sample:

Sample Data

and i want to transform it to this columnar per group.

Output Data

I don't need the 1st column, what I only need is the 2nd columns to be transformed. I already made it thru VBA in excel but it took 2-5 mins to transform 50,000 rows.

I've been self learning python for a while and I hope it will speed up the process thru pandas or numpy.

Thanks a lot.

We have a client who is shipping Subscription products (which are actually Composite products with four to five Bundles of products in them) and they offer delivery on a weekly basis. Their delivery date is always Thursdays. Woocommerce Subscriptions allows for synchronization to a specific date, so we've chosen the "Align Subscription Renewal Day" option and, in a given Product, we've set it to go on Thursdays for each option ("every 4th week", "every 3rd week", etc.)

The caveat with our situation is that orders received the day before (Wednesday) or on the Thursday itself can't be fulfilled that week and need to have their start date/delivery date bumped to the following Thursday. To that end, we've written a function for functions.php using the woocommerce_subscriptions_product_first_renewal_payment_time hook:

I try to understand indexes in Azure Synapse and I'm a bit confused by some of them. 

Regarding the Clustered Columnstore Index, I've a feeling that it works a bit like Apache Parquet, with row groups and column chunks inside. In heap tables the data is not indexed, so it seems pretty clear too.

But what about the clustered and nonclustered indexes? The documentation defines them as:

  Clustered indexes may outperform clustered columnstore tables when a single row needs to be quickly retrieved. For queries where a single or very few row lookup is required to perform with extreme speed, consider a clustered index or nonclustered secondary index. The disadvantage to using a clustered index is that only queries that benefit are the ones that use a highly selective filter on the clustered index column. To improve filter on other columns, a nonclustered index can be added to other columns. However, each index that is added to a table adds both space and processing time to loads.

Here are my questions:

1. Does it mean they're more like the indexes from SQL Server? I mean, the clustered index would order the data by one column and store it as rows? And the non clustered would be an extra sorted index storing only references to the rows?
2. If my assumption about row-based format is correct, does it mean the clustered index is not performant for the analytical queries, doesn't it?
3. What happens if we create a table with both Columnstore and Clustered Indexes? The data is duplicated, once for the columnar format, once for the row format?

Some links I found on that topic, but still have some doubts whether they apply to Synapse:

• https://crmchap.co.uk/understanding-table-distribution-index-types-in-azure-synapse-analytics/
• https://www.sqlservercentral.com/articles/introduction-to-indexes-part-2-%e2%80%93-the-clustered-index
• https://www.sqlservercentral.com/articles/introduction-to-indexes-part-3-%E2%80%93-the-nonclustered-index 
• https://docs.microsoft.com/en-us/sql/t-sql/statements/create-table-azure-sql-data-warehouse?toc=%2Fazure%2Fsynapse-analytics%2Fsql-data-warehouse%2Ftoc.json&bc=%2Fazure%2Fsynapse-analytics%2Fsql-data-warehouse%2Fbreadcrumb%2Ftoc.json&view=azure-sqldw-latest&preserve-view=true#rowstore-table-heap-or-clustered-index

Overview

This code utilises a cluster function that operates on one dimensional arrays and finds the clusters within an array defined by margins to the left and right of every point. I would like to use DBSCAN to replicate this functionality.

Imports:

I have this data frame and table:

Keeping in mind that Redshift is a columnar database server,Lets say I have a table A with 50 columns and I need to join it with table B but I need only 10 columns from table A in my final join result. Lets say table C is a temp table created from Table A with the 10 columns I need.

1. Will [ Table C join Table B] be faster than [Table A join Table B]
2. Assuming Table A was a temporary table itself (derived from other tables), will your response to #1 still hold ?

I need to extract substrings from a file into a new file. Mac or Linux.

The data is between the 4th and 5th "|" symbol.