s3cmd | Chef cookbook for installing and configuring s3cmd | Configuration Management library

This is apparantly not possible with s3cmd. It needs to be done with awscli and the command aws s3 cp test_py3.csv.gz s3://my.bucket/path/ --acl public-read

Fist of all ask 1 question at a time.

For the first one you can simply change the permission with chown like :

The /usr directory contains a variety of system and application software. In particular, the Python binary is typically in /usr/bin/python3 on a Linux system (or container).

Your Kubernetes YAML mounts an emptyDir volume over /usr. That hides everything that was in that directory tree, including the Python binary and all of the Python system libraries. That leads to this error.

Mounting the volume somewhere else would avoid this problem. Containerized applications tend to not be overly picky about "standard" FHS paths, so I might set instead

Using the AWS Command-Line Interface (CLI), you can list the most recent file with:

The easiest method is to define Object Lifecycle Management on the Amazon S3 bucket.

You can specify that objects older than a certain number of days should be expired (deleted). The best part is that this happens automatically on a regular basis and you don't need to run your own script.

If you wanted to do it yourself, the best would be to write a script (eg in Python) to retrieve the list of files and delete ones older than a certain date.

Example: GitHub - jordansissel/s3cleaner: Amazon S3 file cleaner - delete things older than a certain age, matching a pattern, etc.

It's somewhat messier to do as a shell script.

I don't know all the parameters of your task, but I would suggest this scheme:

1. Create SQS queue with 10 mins of visibility delay for messages

2. Subscribe SQS to S3 bucket events, so every object creation event creates message in SQS

3. Subscribe Lambda function to this queue, write code to move the object to your archive location

This is more easily manageable and you don't need a lot of infrastructure to support, since lambda is serverless.

Last time I saw performance comparisons getObjectMetadata was the fastest way to check if an object exists. Using the AWS cli that would be the head-object method, example:

Short Answer

No

Long Answer

When you run docker build the current directory and all of its contents (subdirectories and all) are copied into a staging area called the 'build context'. When you issue a COPY instruction in the Dockerfile, docker will copy from the staging area into a layer in the image's filesystem.

As you can see, this procludes copying files from directories outside the build context.

Workaround

Either download the files you want from their golden-source directly into the image during the build process (this is why you often see a lot of curl statements in Dockerfiles), or you can copy the files (dirs) you need into the build-tree and check them into source control as part of your project. Which method you choose is entirely dependent on the nature of your project and the files you need.

Notes

There are other workarounds documented for this, all of them without exception break the intent of 'portability' of your build. The only quality solutions are those documented here (though I'm happy to add to this list if I've missed any that preserve portability).

Looks like you're not quite sure where your repository code is mounted/placed in your GitHub hosted VM.

From the GitHub Actions Docs:
https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners (the table I copied and pasted doesn't format well below so if you're confused, click on the link)

Filesystems on GitHub-hosted runners GitHub executes actions and shell commands in specific directories on the virtual machine. The file paths on virtual machines are not static. Use the environment variables GitHub provides to construct file paths for the home, workspace, and workflow directories.

Directory Environment variable Description

home HOME Contains user-related data. For example, this directory could contain credentials from a login attempt.

workspace GITHUB_WORKSPACE Actions and shell commands execute in this directory. An action can modify the contents of this directory, which subsequent actions can access.

I just glanced at s3cmd --help