kandi background

incubator-superset | Apache Superset is a Data Visualization | Data Visualization library

by apache Python Version: 0.38.0 License: Apache-2.0

by apache Python Version: 0.38.0 License: Apache-2.0

Download this library from

kandi X-RAY | incubator-superset Summary

incubator-superset is a Python library typically used in Analytics, Data Visualization applications. incubator-superset has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install incubator-superset' or download it from GitHub, PyPI.

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -→.

Support

Support

Quality

Quality

Security

Security

License

License

Reuse

Reuse

kandi-support Support

incubator-superset has a medium active ecosystem.
It has 31662 star(s) with 6530 fork(s). There are 1312 watchers for this library.
It had no major release in the last 12 months.
There are 393 open issues and 4913 have been closed. On average issues are closed in 103 days. There are 82 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of incubator-superset is 0.38.0

incubator-superset Support

Best in #Data Visualization

Average in #Data Visualization

incubator-superset Support

Best in #Data Visualization

Average in #Data Visualization

quality kandi Quality

incubator-superset has 0 bugs and 0 code smells.

incubator-superset Quality

Best in #Data Visualization

Average in #Data Visualization

incubator-superset Quality

Best in #Data Visualization

Average in #Data Visualization

security Security

incubator-superset has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
incubator-superset code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

incubator-superset Security

Best in #Data Visualization

Average in #Data Visualization

incubator-superset Security

Best in #Data Visualization

Average in #Data Visualization

license License

incubator-superset is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

incubator-superset License

Best in #Data Visualization

Average in #Data Visualization

incubator-superset License

Best in #Data Visualization

Average in #Data Visualization

build Reuse

incubator-superset releases are available to install and integrate.
Deployable package is available in PyPI.
Build file is available. You can build the component from source.
Installation instructions, examples and code snippets are available.
incubator-superset saves you 62371 person hours of effort in developing the same functionality from scratch.
It has 70846 lines of code, 3283 functions and 1365 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

incubator-superset Reuse

Best in #Data Visualization

Average in #Data Visualization

incubator-superset Reuse

Best in #Data Visualization

Average in #Data Visualization

Top functions reviewed by kandi - BETA

Coming Soon for all Libraries!

Currently covering the most popular Java, JavaScript and Python libraries. See a SAMPLE HERE.
kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.

incubator-superset Key Features

Apache Superset is a Data Visualization and Data Exploration Platform

incubator-superset Examples and Code Snippets

See all related Code Snippets

default

Copy

Download

http://www.apache.org/licenses/LICENSE-2.0

No package metadata was found for apache-superset

Copy

Download

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ pip install -r requirements/local.txt

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ pip install -e .

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ superset init
Loaded your LOCAL configuration at [/home/cwj0/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset/superset_config.py]
logging was configured successfully
2022-01-14 08:45:43,385:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-01-14 08:45:43,428:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Syncing role definition
2022-01-14 08:45:48,268:INFO:superset.security.manager:Syncing role definition
Syncing Admin perms
2022-01-14 08:45:48,296:INFO:superset.security.manager:Syncing Admin perms
Syncing Alpha perms
2022-01-14 08:45:48,538:INFO:superset.security.manager:Syncing Alpha perms
Syncing Gamma perms
2022-01-14 08:45:48,784:INFO:superset.security.manager:Syncing Gamma perms
Syncing granter perms
2022-01-14 08:45:49,031:INFO:superset.security.manager:Syncing granter perms
Syncing sql_lab perms
2022-01-14 08:45:49,270:INFO:superset.security.manager:Syncing sql_lab perms
Fetching a set of all perms to lookup which ones are missing
2022-01-14 08:45:49,498:INFO:superset.security.manager:Fetching a set of all perms to lookup which ones are missing
Creating missing datasource permissions.
2022-01-14 08:45:49,723:INFO:superset.security.manager:Creating missing datasource permissions.
Creating missing database permissions.
2022-01-14 08:45:49,752:INFO:superset.security.manager:Creating missing database permissions.
Cleaning faulty perms
2022-01-14 08:45:49,756:INFO:superset.security.manager:Cleaning faulty perms

I have mysql and apache superset setup on dockers and connected by a bridge network, what will theSQLAlchemy URI be?

Copy

Download

 Plugin caching_sha2_password could not be loaded:

from sqlalchemy import create_engine
engine = create_engine('mysql://root:sample@172.19.0.5/mysql')
engine.connect()

   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1045, 'Plugin caching_sha2_password could not be loaded: /usr/lib/x86_64-linux-gnu/mariadb19/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory')
    (Background on this error at: http://sqlalche.me/e/13/e3q8)

docker run --detach --network="incubator-superset_default" --name=mysql --env="MYSQL_ROOT_PASSWORD=sample" --publish 3306:3306 mysql --default-authentication-plugin=mysql_native_password

mysql://root:sample@mysql/mysql
mysql://root:sample@172.19.0.5/mysql

-----------------------

 Plugin caching_sha2_password could not be loaded:

from sqlalchemy import create_engine
engine = create_engine('mysql://root:sample@172.19.0.5/mysql')
engine.connect()

   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1045, 'Plugin caching_sha2_password could not be loaded: /usr/lib/x86_64-linux-gnu/mariadb19/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory')
    (Background on this error at: http://sqlalche.me/e/13/e3q8)

docker run --detach --network="incubator-superset_default" --name=mysql --env="MYSQL_ROOT_PASSWORD=sample" --publish 3306:3306 mysql --default-authentication-plugin=mysql_native_password

mysql://root:sample@mysql/mysql
mysql://root:sample@172.19.0.5/mysql

-----------------------

 Plugin caching_sha2_password could not be loaded:

from sqlalchemy import create_engine
engine = create_engine('mysql://root:sample@172.19.0.5/mysql')
engine.connect()

   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1045, 'Plugin caching_sha2_password could not be loaded: /usr/lib/x86_64-linux-gnu/mariadb19/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory')
    (Background on this error at: http://sqlalche.me/e/13/e3q8)

docker run --detach --network="incubator-superset_default" --name=mysql --env="MYSQL_ROOT_PASSWORD=sample" --publish 3306:3306 mysql --default-authentication-plugin=mysql_native_password

mysql://root:sample@mysql/mysql
mysql://root:sample@172.19.0.5/mysql

-----------------------

 Plugin caching_sha2_password could not be loaded:

from sqlalchemy import create_engine
engine = create_engine('mysql://root:sample@172.19.0.5/mysql')
engine.connect()

   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1045, 'Plugin caching_sha2_password could not be loaded: /usr/lib/x86_64-linux-gnu/mariadb19/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory')
    (Background on this error at: http://sqlalche.me/e/13/e3q8)

docker run --detach --network="incubator-superset_default" --name=mysql --env="MYSQL_ROOT_PASSWORD=sample" --publish 3306:3306 mysql --default-authentication-plugin=mysql_native_password

mysql://root:sample@mysql/mysql
mysql://root:sample@172.19.0.5/mysql

-----------------------

 Plugin caching_sha2_password could not be loaded:

from sqlalchemy import create_engine
engine = create_engine('mysql://root:sample@172.19.0.5/mysql')
engine.connect()

   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1045, 'Plugin caching_sha2_password could not be loaded: /usr/lib/x86_64-linux-gnu/mariadb19/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory')
    (Background on this error at: http://sqlalche.me/e/13/e3q8)

docker run --detach --network="incubator-superset_default" --name=mysql --env="MYSQL_ROOT_PASSWORD=sample" --publish 3306:3306 mysql --default-authentication-plugin=mysql_native_password

mysql://root:sample@mysql/mysql
mysql://root:sample@172.19.0.5/mysql

How to configure Celery Worker and Beat for Email Reporting in Apache Superset running on Docker?

Copy

Download

celery worker --app=superset.tasks.celery_app:app --pool=prefork -O fair -c 4 -B

-----------------------

REDIS_HOST = get_env_variable("REDIS_HOST")
REDIS_PORT = get_env_variable("REDIS_PORT")

class CeleryConfig(object):
    BROKER_URL = "redis://%s:%s/0" % (REDIS_HOST, REDIS_PORT)
    CELERY_IMPORTS = (
        'superset.sql_lab',
        'superset.tasks',
    )
    CELERY_RESULT_BACKEND = "redis://%s:%s/1" % (REDIS_HOST, REDIS_PORT)
    CELERY_ANNOTATIONS = {
        'sql_lab.get_sql_results': {
            'rate_limit': '100/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }
    CELERY_TASK_PROTOCOL = 1
    CELERYBEAT_SCHEDULE = {
        'email_reports.schedule_hourly': {
            'task': 'email_reports.schedule_hourly',
            'schedule': crontab(minute='1', hour='*'),
        },
    }

  superset-worker:
    build: *superset-build
    command: ["celery", "worker", "--app=superset.tasks.celery_app:app", "-Ofair"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  superset-beat:
    build: *superset-build
    command: ["celery", "beat", "--app=superset.tasks.celery_app:app", "--pidfile=", "-f", "/app/celery_beat.log"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    volumes: *superset-volumes

-----------------------

REDIS_HOST = get_env_variable("REDIS_HOST")
REDIS_PORT = get_env_variable("REDIS_PORT")

class CeleryConfig(object):
    BROKER_URL = "redis://%s:%s/0" % (REDIS_HOST, REDIS_PORT)
    CELERY_IMPORTS = (
        'superset.sql_lab',
        'superset.tasks',
    )
    CELERY_RESULT_BACKEND = "redis://%s:%s/1" % (REDIS_HOST, REDIS_PORT)
    CELERY_ANNOTATIONS = {
        'sql_lab.get_sql_results': {
            'rate_limit': '100/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }
    CELERY_TASK_PROTOCOL = 1
    CELERYBEAT_SCHEDULE = {
        'email_reports.schedule_hourly': {
            'task': 'email_reports.schedule_hourly',
            'schedule': crontab(minute='1', hour='*'),
        },
    }

  superset-worker:
    build: *superset-build
    command: ["celery", "worker", "--app=superset.tasks.celery_app:app", "-Ofair"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  superset-beat:
    build: *superset-build
    command: ["celery", "beat", "--app=superset.tasks.celery_app:app", "--pidfile=", "-f", "/app/celery_beat.log"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    volumes: *superset-volumes

See all related Code Snippets

Community Discussions

Trending Discussions on incubator-superset

No package metadata was found for apache-superset
I have mysql and apache superset setup on dockers and connected by a bridge network, what will theSQLAlchemy URI be?
Has anyone tried making custom charts with Superset as per preset.io blog?
How to configure Celery Worker and Beat for Email Reporting in Apache Superset running on Docker?
Terraform: Deploying a Docker Compose app on EKS/ECS
how to give individual schema access to users in apache-superset?

Trending Discussions on incubator-superset

QUESTION

No package metadata was found for apache-superset

Asked 2022-Jan-14 at 00:53

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ superset init
Traceback (most recent call last):
  File "/home/cwj0/anaconda3/envs/superset/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
  File "/home/cwj0/anaconda3/envs/superset/bin/superset", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/home/cwj0/anaconda3/envs/superset/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 951, in distribution
    return Distribution.from_name(distribution_name)
  File "/home/cwj0/anaconda3/envs/superset/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 542, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: No package metadata was found for apache-superset

superset db migrateand superset db upgrade also the same result.What is the reason for this? And how should it be solved?Thanks！

ANSWER

Answered 2022-Jan-14 at 00:53

Solved. Because this is the need to refresh the permissions after I add a new page, I don't know why the dependencies are missing, and the installation is solved.

pip install -r requirements/local.txt，pip install -e .

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ pip install -r requirements/local.txt

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ pip install -e .

(superset) cwj0@ubuntu:~/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset$ superset init
Loaded your LOCAL configuration at [/home/cwj0/anaconda3/envs/superset/lib/python3.7/site-packages/incubator-superset/superset_config.py]
logging was configured successfully
2022-01-14 08:45:43,385:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-01-14 08:45:43,428:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Syncing role definition
2022-01-14 08:45:48,268:INFO:superset.security.manager:Syncing role definition
Syncing Admin perms
2022-01-14 08:45:48,296:INFO:superset.security.manager:Syncing Admin perms
Syncing Alpha perms
2022-01-14 08:45:48,538:INFO:superset.security.manager:Syncing Alpha perms
Syncing Gamma perms
2022-01-14 08:45:48,784:INFO:superset.security.manager:Syncing Gamma perms
Syncing granter perms
2022-01-14 08:45:49,031:INFO:superset.security.manager:Syncing granter perms
Syncing sql_lab perms
2022-01-14 08:45:49,270:INFO:superset.security.manager:Syncing sql_lab perms
Fetching a set of all perms to lookup which ones are missing
2022-01-14 08:45:49,498:INFO:superset.security.manager:Fetching a set of all perms to lookup which ones are missing
Creating missing datasource permissions.
2022-01-14 08:45:49,723:INFO:superset.security.manager:Creating missing datasource permissions.
Creating missing database permissions.
2022-01-14 08:45:49,752:INFO:superset.security.manager:Creating missing database permissions.
Cleaning faulty perms
2022-01-14 08:45:49,756:INFO:superset.security.manager:Cleaning faulty perms

Source https://stackoverflow.com/questions/70694733

QUESTION

I have mysql and apache superset setup on dockers and connected by a bridge network, what will theSQLAlchemy URI be?

Asked 2020-Dec-11 at 10:09

I pulled the official superset image:

git clone https://github.com/apache/incubator-superset.git

then added the MYSQL Client to requirements.txt

cd incubator-superset
touch ./docker/requirements-local.txt
echo "mysqlclient==1.4.6" >> ./docker/requirements-local.txt
docker-compose build --force-rm
docker-compose up -d

After which I made the MYSQL Container

docker run --detach --network="incubator-superset_default" --name=vedasupersetmysql --env="MYSQL_ROOT_PASSWORD=vedashri" --publish 6603:3306 mysql

Then connected Mysql to the Superset Bridge.

The bridge network is as follows:

docker inspect incubator-superset_default
[
    {
        "Name": "incubator-superset_default",
        "Id": "56db7b47ecf0867a2461dddb1219c64c1def8cd603fc9668d80338a477d77fdb",
        "Created": "2020-12-08T07:38:47.94934583Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "07a6e0d5d87ea3ccb353fa20a3562d8f59b00d2b7ce827f791ae3c8eca1621cc": {
                "Name": "superset_db",
                "EndpointID": "0dd4781290c67e3e202912cad576830eddb0139cb71fd348019298b245bc4756",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "096a98f22688107a689aa156fcaf003e8aaae30bdc3c7bc6fc08824209592a44": {
                "Name": "superset_worker",
                "EndpointID": "54614854caebcd9afd111fb67778c7c6fd7dd29fdc9c51c19acde641a9552e66",
                "MacAddress": "02:42:ac:13:00:05",
                "IPv4Address": "172.19.0.5/16",
                "IPv6Address": ""
            },
            "34e7fe6417b109fb9af458559e20ce1eaed1dc3b7d195efc2150019025393341": {
                "Name": "superset_init",
                "EndpointID": "49c580b22298237e51607ffa9fec56a7cf155065766b2d75fecdd8d91d024da7",
                "MacAddress": "02:42:ac:13:00:06",
                "IPv4Address": "172.19.0.6/16",
                "IPv6Address": ""
            },
            "5716e0e644230beef6b6cdf7945f3e8be908d7e9295eea5b1e5379495817c4d8": {
                "Name": "superset_app",
                "EndpointID": "bf22dab0714501cc003b1fa69334c871db6bade8816724779fca8eb81ad7089d",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "b09d2808853c54f66145ac43bfc38d4968d28d9870e2ce320982dd60968462d5": {
                "Name": "superset_node",
                "EndpointID": "70f00c6e0ebf54b7d3dfad1bb8e989bc9425c920593082362d8b282bcd913c5d",
                "MacAddress": "02:42:ac:13:00:07",
                "IPv4Address": "172.19.0.7/16",
                "IPv6Address": ""
            },
            "d08f8a2b090425904ea2bdc7a23b050a1327ccfe0e0b50360b2945ea39a07172": {
                "Name": "superset_cache",
                "EndpointID": "350fd18662e5c7c2a2d8a563c41513a62995dbe790dcbf4f08097f6395c720b1",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            },
            "e21469db533ad7a92b50c787a7aa026e939e4cf6d616e3e6bc895a64407c1eb7": {
                "Name": "vedasupersetmysql",
                "EndpointID": "d658c0224d070664f918644584460f93db573435c426c8d4246dcf03f993a434",
                "MacAddress": "02:42:ac:13:00:08",
                "IPv4Address": "172.19.0.8/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "incubator-superset",
            "com.docker.compose.version": "1.26.0"
        }
    }
]

How should I form the SQLAlchemy URI? I have tried mysql://user:password@8088:6603/database-name

But it shows connection error, when I enter this URI.

If there is any related documentation, that would also help.

ANSWER

Answered 2020-Dec-08 at 18:54

I don't have a lot of experience with Docker, but I don't think you should use 8088 as the host for your MySQL database.

Try using mysql://user:password@172.19.0.8:6603/database-name as the URI.

Source https://stackoverflow.com/questions/65198052

QUESTION

How to configure Celery Worker and Beat for Email Reporting in Apache Superset running on Docker?

Asked 2020-May-12 at 11:32

I am running Superset via Docker. I enabled the Email Report feature and tried it:

However, I only receive the test email report. I don't receive any emails after.

This is my CeleryConfig in superset_config.py:

class CeleryConfig(object):
    BROKER_URL = 'sqla+postgresql://superset:superset@db:5432/superset'
    CELERY_IMPORTS = (
        'superset.sql_lab',
        'superset.tasks',
    )
    CELERY_RESULT_BACKEND = 'db+postgresql://superset:superset@db:5432/superset'
    CELERYD_LOG_LEVEL = 'DEBUG'
    CELERYD_PREFETCH_MULTIPLIER = 10
    CELERY_ACKS_LATE = True
    CELERY_ANNOTATIONS = {
        'sql_lab.get_sql_results': {
            'rate_limit': '100/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }
    CELERYBEAT_SCHEDULE = {
        'email_reports.schedule_hourly': {
            'task': 'email_reports.schedule_hourly',
            'schedule': crontab(minute=1, hour='*'),
        },
    }

The documentation says I need to run the celery worker and beat.

celery worker --app=superset.tasks.celery_app:app --pool=prefork -O fair -c 4
celery beat --app=superset.tasks.celery_app:app

I added them to the 'docker-compose.yml':

superset-worker:
    build: *superset-build
    command: >
      sh -c "celery worker --app=superset.tasks.celery_app:app -Ofair -f /app/celery_worker.log &&
             celery beat --app=superset.tasks.celery_app:app -f /app/celery_beat.log"
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    volumes: *superset-volumes

Celery Worker is indeed working when sending the first email. The log file is also visible. However, the celery beat seems to not be functioning. There is also no 'celery_beat.log' created.

If you'd like a deeper insight, here's the commit with the full implementation of the functionality.

How do I correctly configure celery beat? How can I debug this?

ANSWER

Answered 2020-Apr-21 at 14:06

I believe Celery needs to run inside your superset container - so you'll need to modify your dockerfile and entrypoint.
BUT you should really first daemonize celery so you don't have to monitor and restart celery [see how to detect failure and auto restart celery worker and http://docs.celeryproject.org/en/latest/userguide/daemonizing.html].
See an example here for how to run a daemonized celery process in docker: Docker - Celery as a daemon - no pidfiles found

Source https://stackoverflow.com/questions/61331079

QUESTION

Terraform: Deploying a Docker Compose app on EKS/ECS

Asked 2020-Apr-21 at 15:31

TL;DR

I use an open-source server application running on Docker Compose. It has a few services, including PostgreSQL DB and Redis.

How can I best deploy this application to AWS in full IaC with Terraform?

Solutions so far

1. AWS `ecs-cli`

ecs-cli now supports sending docker compose configs in Amazon ECS.

However, I do not think it could be integrated with the Terraform workflow (which is maybe not a big fuss). What I know for sure is that ecs-cli is not supported in CloudFormation, as per this issue, still open at this time. So I assume it cannot easily be added to Terraform either.

2. The hard EKS way

Take your docker-compose.yml file, translate it to kubectl YAML.
(Prepare to doing so every time the package upgrades).
Deploy using Terraform's + EKS APIs (minimal example).

But that is not fully IaC yet. And you have to retranslate your config each time the docker-compose changes in the source repository. And it sounds like a lot of work.

3. Using a Helm chart

Write a Helm chart for the application.
Run Terraform to start a cluster.
Still run Terraform with a helm provider to install the application with Helm on the cluster.

4. [Not OK] k8s Kompose

I read Kompose can automagically translate a Docker Compose configuration to a k8s configuration, but they don't appear to be ported on AWS, not to talk about Terraform.

5. [Not OK] The dirty AMI solution

Build a custom EC2 AMI with Packer.
Using Terraform, set up the DB, Redis and all permissions / network / etc.
Start an EC2 instance with the custom AMI.
The AMI contains (customized) application code, especially the docker-compose.yml. And the Docker images.
The AMI starts a Docker Compose systemctl service.

That would kind of hurt: long builds, difficult monitoring, no scaling.

Side notes

I mentioned I need full IaC. What I mean by it is:
1. The written config (and it only) on master tells you what is deployed.
2. The deploy will complete without my team having to run any other command. It basically works either on CI/CD or on single-push'n'go.
3. Monitoring and alerting are easily configured in the IaC.
I wish to replace some services with AWS components (namely, the PostgreSQL service by an RDS, and the Redis service with an ElastiCache).
The application happens to be Apache Superset. However I am also wondering what is considered the best general approach to this Docker Compose problem.

ANSWER

Answered 2020-Apr-20 at 14:22

1. Wait and See

Who knows, ecs-cli-v2 might be better integrated with CloudFormation and/or Terraform.

2. Use a Helm chart

As mentioned in the question. Probably the best solution, albeit requiring a (little) effort to parametrize Helm.

See also: Getting started with Helm.

3. Docker Swarm + CloudFormation + Terraform

Docker Swarm now accepts inputs from a docker-compose.yml file. The template can be found and configured here. Once configured, it may be integrated to a Terraform infrastructure.

This (3-year old) tutorial explains how to use Docker Swarm mode on AWS.

To launch the container, if necessary (not fully investigated, feedback is welcome), you could use Terraform's local-exec. This way you can SSH into the master node and run docker stack deploy and other similar commands, while still having all written down in IaC style.

Source https://stackoverflow.com/questions/61122181

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install incubator-superset

[See in the documentation](https://superset.incubator.apache.org/docs/installation/installing-superset-using-docker-compose).

Support

Superset speaks many SQL dialects through SQLAlchemy - a Python SQL toolkit that is compatible with most databases. Here are some of the major database solutions that are supported:. <p align="center"> <img src="superset-frontend/images/redshift.png" alt="redshift" border="0" width="106" height="41"/> <img src="superset-frontend/images/google-biquery.png" alt="google-biquery" border="0" width="114" height="43"/> <img src="superset-frontend/images/snowflake.png" alt="snowflake" border="0" width="152" height="46"/> <img src="superset-frontend/images/presto.png" alt="presto" border="0" width="152" height="46"/> <img src="superset-frontend/images/druid.png" alt="druid" border="0" width="135" height="37" /> <img src="superset-frontend/images/postgresql.png" alt="postgresql" border="0" width="132" height="81" /> <img src="superset-frontend/images/mysql.png" alt="mysql" border="0" width="119" height="62" /> <img src="superset-frontend/images/mssql-server.png" alt="mssql-server" border="0" width="93" height="74" /> <img src="superset-frontend/images/db2.png" alt="db2" border="0" width="62" height="62" /> <img src="superset-frontend/images/sqlite.png" alt="sqlite" border="0" width="102" height="45" /> <img src="superset-frontend/images/sybase.png" alt="sybase" border="0" width="128" height="47" /> <img src="superset-frontend/images/mariadb.png" alt="mariadb" border="0" width="83" height="63" /> <img src="superset-frontend/images/vertica.png" alt="vertica" border="0" width="128" height="40" /> <img src="superset-frontend/images/oracle.png" alt="oracle" border="0" width="121" height="66" /> <img src="superset-frontend/images/firebird.png" alt="firebird" border="0" width="86" height="56" /> <img src="superset-frontend/images/greenplum.png" alt="greenplum" border="0" width="140" height="45" /> <img src="superset-frontend/images/clickhouse.png" alt="clickhouse" border="0" width="133" height="34" /> <img src="superset-frontend/images/exasol.png" alt="exasol" border="0" width="106" height="59" /> <img src="superset-frontend/images/monet-db.png" alt="monet-db" border="0" width="106" height="46" /> <img src="superset-frontend/images/apache-kylin.png" alt="apache-kylin" border="0" width="56" height="64"/> </p>. A complete list of supported databases can be found [here](https://superset.incubator.apache.org/docs/databases/installing-database-drivers).

DOWNLOAD this Library from

Reuse Trending Solutions

14 best Python Telegram Bot

21 best C# Game Development

Build AI Fake News Detector

Build Text Summarizer in Python

Getting Started with Web3

24 best Python Blockchain

26 best Python Face Recognition

Build Ludo Game in Python

Build Location Based AR App

Build AI Study Assistant Bot

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items

Find more libraries

Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Explore Related Topics

Data Visualization

Share this Page

Reuse Pre-built Kits with incubator-superset

Build Energy Analytics Solution

VIT Hackathon Starter Kit

IoT Energy Efficiency Management Ki

IoT energy management Kit

nithyavarshini23 Kit

See all related Kits

Reuse Analytics Kits

Build AI Course Recommender

Refugee Support Solutions

BuildWithAI Challenge 1

Student Well Beings

Impact of COVID 19 on mental health

See all related Kits

Consider Popular Data Visualization Libraries

d3

by d3

echarts

by apache

algorithm-visualizer

by algorithm-visualizer

incubator-superset

by apache

sheetjs

by SheetJS

See all Data Visualization Libraries

Try Top Libraries by apache

echarts

by apache TypeScript

superset

by apache TypeScript

dubbo

by apache Java

spark

by apache Scala

airflow

by apache Python

See all Libraries by this author

Compare Data Visualization Libraries with Highest Support

matplotlib

by matplotlib

ipython

by ipython

echarts

by apache

redash

by getredash

d3

by d3

See all Data Visualization Libraries

Compare Data Visualization Libraries with Highest Quality

VWorkflows

by miho

molgenis

by molgenis

SensorWebClient

by 52North

zkspreadsheet

by zkoss

goffish

by usc-cloud

See all Data Visualization Libraries

Compare Data Visualization Libraries with Highest Security

TimeBench

by ieg-vienna

VWorkflows

by miho

molgenis

by molgenis

SensorWebClient

by 52North

GoogleFormUploadExample

by FoamyGuy

See all Data Visualization Libraries

Compare Data Visualization Libraries with Permissive License

TimeBench

by ieg-vienna

music-visualization

by michaelbrooks

prov-viewer

by gems-uff

first-stories-twitter

by mvogiatzis

MGA

by crukci-bioinformatics

See all Data Visualization Libraries

Compare Data Visualization Libraries with Highest Reuse

RBHC

by intuit

corella

by nk412

webviz-config

by equinor

mplhep

by scikit-hep

pyworkout-toolkit

by triskadecaepyon

See all Data Visualization Libraries

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items

Find more libraries

Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

© 2022 Open Weaver Inc.
Terms
Privacy policy
Contact us

© 2022 Open Weaver Inc.