clearml | Magical CI/CD to streamline your ML workflow | Machine Learning library

i would recommend you to check out the Storage gateway S3 behind the gateway you can use the NFS, EFS or S3 bucket.

Read more at : https://aws.amazon.com/storagegateway/?whats-new-cards.sort-by=item.additionalFields.postDateTime&whats-new-cards.sort-order=desc

There are multiple ways you can do this. You can also use the CSI driver to connect the S3 also.

https://github.com/ctrox/csi-s3

rclone is nice option if you can use it, which will sync data to the POD host system in that if large files are there it might take time due to file size and network latecy.

Personal suggestion S3 is object storage so if you are looking forward to do file operations like writing the file or zip file it might take time to do operation based on my personal experience.

Remember that s3 is NOT a file system, but an object store - while mounting IS an incredibly useful capability - I wouldn't leverage anything more than file read or create - don't try to append a file, don't try to use file system trickery

If that the case I would recommend using the NFS or SSD to the container.

while if we look for s3fs-fuse it has own benefit of multipart upload and MD5 & local caching etc.

The easiest way you can write your own script which will sync to the local directory with the directory of S3 bucket over HTTP or else Storage gateway S3 is good option.

Amazon S3 File Gateway provides a seamless way to connect to the cloud in order to store application data files and backup images as durable objects in Amazon S3 cloud storage. Amazon S3 File Gateway offers SMB or NFS-based access to data in Amazon S3 with local caching.

Disclaimer: I'm part of the ClearML (formerly Trains) Team

I think this screenshot is taken from the premium version... The feature itself exists in the open-source version, but I "think" some of the dataset visualization capabilities are not available in the open-source self hosted version.

Nonetheless, you have a fully featured feature-store, with the ability to add your own metrics / samples for every dataset/feature version. The open-source version also includes the advanced versioning & delta based storage for datasets/features (i.e. only the change set from the parent version is stored)

Disclaimer I'm part of the ClearML (formerly Trains) team.

To get an existing Task object for a running (or completed/failed) experiment, assuming we know Task ID:

Disclaimer I'm part of the ClearML (formerly Trains) team.

pytorch_lightning is creating a new Tensorboard for each experiment. When ClearML logs the TB scalars, and it captures the same scalar being re-sent again, it adds a prefix so if you are reporting the same metric it will not overwrite the previous one. A good example would be reporting loss scalar in the training phase vs validation phase (producing "loss" and "validation:loss"). It might be the task.close() call does not clear the previous logs, so it "thinks" this is the same experiment, hence adding the prefix networkB to the loss. As long as you are closing the Task after training is completed you should have all experiments log with the same metric/variant (title/series). I suggest opening a GitHub issue, this should probably be considered a bug.

Disclaimer I'm part of the ClearML (formerly Trains) team.

To set credentials (and clearml-server hosts) you can use Task.set_credentials. To specify the S3 bucket as output for all artifacts (and debug images for that matter) you can just set it as the files_server.

For example:

Disclaimer: I'm a member of the ClearML team (formerly Trains)

In ClearML, debug images' URL is registered once they are uploaded to the fileserver. The WebApp doesn't actually decide on the URL for each debug image, but rather obtains it for each debug image from the server. This allows you to potentially upload debug images to a variety of storage targets, ClearML File Server simply being the most convenient, built-in option.

So, the WebApp will always look for localhost:8008 for debug images that have already been uploaded to the fileserver and contain localhost:8080 in their URL. A possible solution is to simply add another tunnel in the form of ssh -N -L 8081:127.0.0.1:8081 user@hostA.

For future experiments, you can choose to keep using 8081 (and keep using this new tunnel), or to change the default fileserver URL in clearml.conf to point to port localhost:18081, assuming you're running your experiments from the same machine where the tunnel to 18081 exists.

Disclaimer: I'm a ClearML (Trains) team member

Basically the docker-compose will expose only the API/Web/File server , you can further limit the exposure to your localhost only, by changing the following section in your ClearML server docker-compose.yml

Disclaimer: I'm a member of the ClearML team (formerly Trains)

I would really like to bypass this restriction and run experiments on my local machine, not connecting to any remote destination.

A few options:

1. The Clearml Free trier offers free hosting for your experiments, these experiment are only accessible to you, unless you specifically want to share them among your colleagues. This is probably the easiest way to get started.
2. Install the ClearML-Server basically all you need is docker installed and you should be fine. There are full instructions here , this is the summary: