kafka-topic-dumper | Python tool to get messages | Cloud Storage library

I am using Nifi to send data from Pub/Sub queue to Cloud storage. I'm using the ConsumeGCPubSub processor to fetch data from the queue and the PutGCSObject processor to add Cloud Storage in Nifi. But the PutGCSObject processor is sending duplicate data in Cloud Storage.

I also see that this data has the same MD5 Hash code in its Cloud Storage records. What could be causing this and how can I fix it?

I double checked:

• pub/sub messages is not duplicated.
• When I send 30 piece of data, there are come exactly 30 pieces in Nifi
• I checked my google storage have different data. But there was not..
• When I examine it, the number of data coming from the queue and exiting the PutGCSObject processor as success is the same, but I see that the data is written over and over again. When I looked into NiFi Data Provenance, I found that there are multiple data with the same FlowFile UUID.

The questions are related to snowflake account with organization/ orgadmin role enabled.

1.Is it possible to detach a snowflake account from a snowflake organization?

a. If yes, will the removed account become a standalone(separate contract) account? How does the billing work?

b. Will the account url change after detachment?

c. Procedure to achieve the above?

1. In an organization, are the background services charged on each account?
2. Can we clone a database across accounts within an organization?
3. What happens to the other accounts within an organization when the primary account is deleted?
4. Can we get a cost comparison table between 2 standalone accounts and an organization with 2 accounts?
5. After detachment can the account type/region/cloud provider be changed?

I have asked similar questions to snowflake support through support ticket system, But would like to get answers from the community too.

P.S If I get an answer from Snowflake, I will post it here!.

Due to the nature of our business, we basically need to disclose where in the globe the files uploaded by our users are located.

In other words, we need the exact address where the data storage that keeps these files is located.

We're using Google Firebase's Cloud Storage and, even though they mention which city each location option refers to, we are unable to check the exact address.

The bucket that corresponds to our Google Cloud Storage is currently configured as: us (multiple regions in United States), which I suppose makes it even worse to pinpoint where the data resides. But that is an easy fix: we can simply start from scratch selecting a specific region as our storage location.

The main issue, however, is that, even if selecting a specific location, we can't really know the address where those files will be stored.

Has anyone ever come across something like this?

I tried getting support in my project's Google Cloud Platform, but apparently I need to purchase it. And I'm afraid that they won't be able to help me.

In case someone has contacted their support and got this answer, please let me know.

I have 10 requests per second of data I want to save that looks like the entry below. I need to save this data after a CloudRun function completes. (My infrastructure is on google-cloud-platform). The data will be used as a data set for machine learning.

I have stored thousands of images in GCP Cloud Storage in very high resolution. I want to serve these images in an iOS/Android App and on a website. I don't want to serve all the time the high-resolution version and wondered whether I have to create duplicate images in different resolutions - which seems very inefficient. The perfect solution would be that I can append a parameter like ?size=100 to the image URL. Is something like that natively possible with GCP Cloud Storage?

I don't find anything in the documentation from cloud storage: https://cloud.google.com/storage/docs. Several other resources link to deprecated solutions: https://medium.com/google-cloud/uploading-resizing-and-serving-images-with-google-cloud-platform-ca9631a2c556

What is the best solution to implement such functionality?

I was wondering that how do storage solutions like S3 or Google Drive check whether their storage platform is being abused for the storage of malicious content?

e.g. if someone uploads a password protected zip file to their servers, I don't see a way on how they can verify it. For unencrypted files, I can understand some sort of file parser could work. But if someone uploads a password protected file, the only way to see/verify the contents is try to brute force your way into it (ignoring the moral obligations for the organisation to not do that).

So, how do these companies/solutions verify the kind of data that is being uploaded on their platforms?

I want to find a free cloud storage service with free API, that could help me back up some files automatically.

I want to write some script (for example python) to upload files automatically.

I investigated OneDrive and GoogleDrive. OneDrive API is not free, GoogleDrive API is free while it need human interactive authorization before using API.

For now I'm simply using email SMTP protocol to send files as email attachments, but there's a max file size limition, which will fail me in the future, as my file size is growing.

Is there any other recommendations ?

As an intro, I'm developing an app with Flutter that has an audio section.

I would like to address two subjects.

1. For the moment the audio is stored in the cloud, more specific using Firebase. The main problem is that the pricing is not very supportive when the bandwidth threshold is exceeded. Also, as I discovered, each song is downloaded completely when trying to play it. Therefore, it doesn't matter that I want play 10 seconds or 1 minute of a song, the same traffic is generated. I'm using just_audio package as audio library and I'm wondering if there is a solution to integrate a stream base solution that implies buffering.

2. As I've seen in the debug logs, a HTTP request is sent every time a song (from the cloud) is requested to play. Now, my concerns are that I can't use just_audio for streaming. Is there a cloud solution that fulfills a good compromise between price and bandwidth, even if the song is downloaded entirely each time the play action is required? I'm taking in consideration to develop an offline mode for the audio section, so that each song could be played from the local memory. Even so, it must be a user option, not a by default feature.

I'm using @google-cloud/storage package and generating signed url to upload file like this: