unzip | Tiny unzip helper class | DevOps library

Recently I face an issues to install my dependencies using latest Node and NPM on my MacBook Air M1 machine. Then I found out M1 is not supported latest Node version. So my solution, to using NVM and change them to Node v14.16

Everything works well, but when our team apply new eslint configuration. Yet, I still not sure whether eslint was causes the error or not.

Scenario

For a map tiling project (think google earth) I have a .mbtile file that I know nothing about other than it has a lot of data about cities on the planet. I've been picking at it for a couple days and have found out the following:

• the file is a sqlite database
• the database has a table tiles which is full of blobs
• the blobs are binary with gzip signatures
• after unzipping, the result is a protocol buffer

I haven't worked with protobuf much so I'm getting a crash course. I took the unzipped binary and with protoc --decode_raw received the following

I've got a docker image running 8.0 and want to upgrade to 8.1. I have updated the image to run with PHP 8.1 and want to update the dependencies in it.

The new image derives from php:8.1.1-fpm-alpine3.15

I've updated the composer.json and changed require.php to ^8.1 but ran into the following message when running composer upgrade:

I am getting Partial credentials found in env error while running below command.

aws sts assume-role-with-web-identity --role-arn $AWS_ROLE_ARN --role-session-name build-session --web-identity-token $BITBUCKET_STEP_OIDC_TOKEN --duration-seconds 1000

I am using below AWS CLI and Python version-

In this programming problem, the input is an n×m integer matrix. Typically, n≈ 10⁵ and m ≈ 10. The official solution (1606D, Tutorial) is quite imperative: it involves some matrix manipulation, precomputation and aggregation. For fun, I took it as an STUArray implementation exercise.

I have managed to implement it using STUArray, but still the program takes way more memory than permitted (256MB). Even when run locally, the maximum resident set size is >400 MB. On profiling, reading from stdin seems to be dominating the memory footprint:

Functions readv and readv.readInt, responsible for parsing integers and saving them into a 2D list, are taking around 50-70 MB, as opposed to around 16 MB = (10⁶ integers) × (8 bytes per integer + 8 bytes per link).

Is there a hope I can get the total memory below 256 MB? I'm already using Text package for input. Maybe I should avoid lists altogether and directly read integers from stdin to the array. How can we do that? Or, is the issue elsewhere?

I have an installation file for my app and it contains the .exe file and a .zip file as well.

Recently I added a code in the [Code] section that is extracting files from zip, But actually, it happened after the installation is done, and the progress bar is 100%, So what I want is to make that code's (unzipping) process work with the progress bar and show the user what is extracting at the moment.

For example: let's say extracting files will take 50% of the progress bar and the rest will take it the code section while it is unzipping, with the state about what is extracting at the moment.

I have a rather large (1.3 GB, unzipped) csv file, with 2 dense columns and 1.4 K sparse columns, about 1 M rows.

I need to make a pandas.DataFrame from it.

For small files I can simply do:

Does AWS clear the /tmp directory automatically?

Is there a flush() sort of function? If not, how can I remove/delete all the files from the /tmp folder?

I have an AWS Lambda function where I download a file to my /tmp folder. I unzip the zipped file and gzip all individual files. All this happens within the /tmp directory and before I upload the gzipped files to S3 again.

Afterwards, I no longer need the files in my /tmp folder & would like to clear the directory.

If I open /tmp from my local macOS machine, I don't see any related files at all so I am not sure how to check if they are successfully deleted or not.

I have a partitioned table with less than 600MB data in the targeted partitions.

When I run an EXPORT query with a subquery that returns ~1000 rows, EXPORT shards the data into 27(!) files.

This is not a big query result. I assume it's happening because the optimizer sees the 600MB, but I'm not sure.

Has anyone else encountered this? I am GZIPing the results so concatenating would involve unzipping, appending and zipping again...