cloudE | 基于spring cloud的分布式系统架构。提供整套的微服务组件，包括服务发现、服务治理、链路追踪、服务监控等 | Microservice library

When I run queries on external Parquet tables in Snowflake, the queries are orders of magnitude slower than on the same tables copied into Snowflake or with any other cloud data warehouse I have tested on the same files.

Context:

I have tables belonging to the 10TB TPC-DS dataset in Parquet format on GCS and a Snowflake account in the same region (US Central). I have loaded those tables into Snowflake using create as select. I can run TPC-DS queries(here #28) on these internal tables with excellent performance. I was also able to query those files on GCS directly with data lake engines with excellent performance, as the files are "optimally" sized and internally sorted. However, when I query the same external tables on Snowflake, the query does not seem to finish in reasonable time (>4 minutes and counting, as opposed to 30 seconds, on the same virtual warehouse). Looking at the query profile, it seems that the number of records read in the table scans keeps growing indefinitely, resulting in a proportional amount of spilling to disk.

The table happens to be partitioned but it those not matter on the query of interest (which I tested with other engines).

What I would expect:

Assuming proper data "formatting", I would expect no major performance degradation compared to internal tables, as the setup is technically the same - data stored in columnar format in cloud object store - and as it is advertised as such by Snowflake. For example I saw no performance degradation with BigQuery on the exact same experiment.

Other than double checking my setup, I see don't see many things to try...

This is what the "in progress" part of the plan looks like 4 minutes into execution on the external table. All other operators are at 0% progress. You can see external bytes scanned=bytes spilled and 26G!! rows are produced. And this is what it looked like on a finished execution on the internal table executed in ~20 seconds. You can see that the left-most table scan should produce 1.4G rows but had produced 23G rows with the external table.

This is a sample of the DDL I used (I also tested without defining the partitioning column):

I'm currently running PySpark via local mode. I want to be able to efficiently output parquet files to S3 via the S3 Directory Committer. This PySpark instance is using the local disk, not HDFS, as it is being submitted via spark-submit --master local[*].

I can successfully write to my S3 Instance without enabling the directory committer. However, this involves writing staging files to S3 and renaming them, which is slow and unreliable. I would like for Spark to write to my local filesystem as a temporary store, and then copy to S3.

I have the following configuration in my PySpark conf:

TF project:

• main.tf
• inputs.tf

The contents are: main.tf

I have a domain which name is asifulmamun.info

Then, I've purchased a hosting for host website and connect this domain to hosting with cpanel by nameserver change.

I've create an email with this domain from Cpanel i.e. xx@asifulmamun.info.

Hosting provider told me that, my email has a limit for sending or receiving up to (25-30) email per hour.

But, if i will need to send/receive more than email from limitation how can I do this?

I think it's using my hosting server protocol for using mail email service.

Is it possible using another service provider protocol for using more than email from hosting server protocol?

Is it possible to use gmail server without purchase google cloude?

Is it possible, my domain will host in my exist hosting (Cpanel) and mail protocol using another service provider i.e. google, godaddy, aws or any service provider? If possible how?

The spark docmentation suggests using spark-hadoop-cloud to read / write from S3 in https://spark.apache.org/docs/latest/cloud-integration.html .

There is no apache spark published artifact for spark-hadoop-cloud. Then when trying to use the Cloudera published module the following exception occurs

I'm relatively new to Spectron and Jest and I can't figure out why the app isn't launching when I call app.start() in my unit test. Right now when I run npm test, the app won't start, eventually times out (even after 30 seconds) and always sends this error message:

Timeout - Async callback was not invoked within the 15000 ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 15000 ms timeout specified by jest.setTimeout. at mapper (node_modules/jest-jasmine2/build/queueRunner.js:27:45)

So far I've tried:

• making sure I'm using the correct versions of spectron and electron (11.0.0 and 9.0.0 respectively)
• running npm test from my root folder, my src folder, and my tests folder.
• deleting my node_modules folder, reinstalling everything, and rebuilding the app.
• using path.join(__dirname, '../../', 'node_modules', '.bin', 'electron') as my app.path.

Here's my test1.js file:

I'm working on an electron app, using React on the front end and I'm attempting to use Jest for testing. However, when I try to run tests I get the following error:

SyntaxError: C:\Users\JimArmbruster\source\repos\cyborg_cloud_explorer\cyborg_cloud_explorer_gui\src\assets\custom_components\stylesheets\buttons.css: Support for the experimental syntax 'decorators-legacy' isn't currently enabled (1:1):

I desperately try to implement a simple Webhook for my DialogFlow CX agent. Never done this before so I just copy paste the index.js and package.json code I found on the following page to my Google Cloud Function: DialogFlow CX calculate values

But it seems this is not working. When trying to deploy the Cloud Function I get the error "Error: listen EADDRINUSE: address already in use :::8080".

Same happens if I take this sample code: Dialogflow CX webhook for fulfilment to reply user using nodejs

What am I doing wrong? I am editing the code and trying to deploy it directly in the Google Cloude web console and not via a command prompt tool.

HERE SOME MORE DETAILS:

Setup of Google Cloud Function: I set up a new Google Cloud Function via Google Cloud Console by clicking Create Function. I set Region to us-east1, Trigger type to HTTP and Allow unauthenticated invocations. Then I save, update the index.js and package.json as described below and click Deploy. The result is that deployment could not be done because of Error: listen EADDRINUSE: address already in use :::8080.

Here the code I put into to index.js:

I am using JScript to fetch data from a JSON API url. (I have added the data in the JSON file below - These are 8 horse races and each races displays Horse number, Horse name and their odds). I am trying to write a Jscript to display each races on individual table inside a container/DIV. I should be able to place each race on different section of the website. Eg. Race1 on home page on the top, Race2 on Home page in the bottom and Race 3 on another place on the website. With my current code, when I add 2 races or more, only one displays. Please note that I am only beginner in Javascript.

Data from JSON