dagster | orchestration platform for the development production | BPM library

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: authorize-allow-all
  namespace: dagster
spec:
  selector:
      matchLabels:
        component: dagit
  action: ALLOW

022-03-23T15:25:47.112544Z  debug   envoy filter    tls inspector: new connection accepted
2022-03-23T15:25:47.117785Z debug   envoy filter    tls:onServerName(), requestedServerName: radacted
2022-03-23T15:25:47.117863Z debug   envoy conn_handler  [C8274] new connection
2022-03-23T15:25:47.202884Z debug   envoy http  [C8274] new stream
2022-03-23T15:25:47.202965Z debug   envoy http  [C8274][S2813519150144299458] request headers complete (end_stream=false):
‘:authority’, ‘radacted’
‘:path’, ‘/graphql’
‘:method’, ‘POST’
‘dagster-testing’, ‘3’
‘authorization’, ‘Bearer eyJ0...’
‘content-type’, ‘application/json’
‘user-agent’, ‘PostmanRuntime/7.29.0’
‘accept’, ‘*/*’
‘cache-control’, ‘no-cache’
‘postman-token’, ‘d625526b-3f8a-4f9f-9b8d-352a1683b9d5’
‘accept-encoding’, ‘gzip, deflate, br’
‘connection’, ‘keep-alive’
‘content-length’, ‘110’
‘cookie’, ‘_oauth2_proxy_csrf=radacted’
2022-03-23T15:25:47.203065Z debug   envoy router    [C8274][S2813519150144299458] cluster ’outbound|3000||dagster-dagit.dagster.svc.cluster.local’ match for URL ‘/graphql’
2022-03-23T15:25:47.203121Z debug   envoy router    [C8274][S2813519150144299458] router decoding headers:
‘:authority’, ‘radacted’
‘:path’, ‘/graphql’
‘:method’, ‘POST’
‘:scheme’, ‘https’
‘dagster-testing’, ‘3’
‘authorization’, ‘Bearer eyJ0...’
‘content-type’, ‘application/json’
‘user-agent’, ‘PostmanRuntime/7.29.0’
‘accept’, ‘*/*’
‘cache-control’, ‘no-cache’
‘postman-token’, ‘d625526b-3f8a-4f9f-9b8d-352a1683b9d5’
‘accept-encoding’, ‘gzip, deflate, br’
‘content-length’, ‘110’
‘cookie’, ‘_oauth2_proxy_csrf=radacted’
‘x-forwarded-for’, ‘10.11.225.213’
‘x-forwarded-proto’, ‘https’
‘x-envoy-internal’, ‘true’
‘x-request-id’, ‘8aee0b05-65b3-420b-9ea4-2c3c11f3f93b’
‘x-envoy-decorator-operation’, ‘dagster-dagit.dagster.svc.cluster.local:3000/*’
‘x-envoy-peer-metadata’, ‘ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0AUKBkxBQkVMUxLFBSrCBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiEKBWlzdGlvEhgaFmRhZ3N0ZXItaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo1OThiYzU5OGQ5ChIKB3JlbGVhc2USBxoFaXN0aW8KOwofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIYGhZkYWdzdGVyLWluZ3Jlc3NnYXRld2F5Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoRCgNzaGESChoINjE0ZWE5MmMKIgoXc2lkZWNhci5pc3Rpby5pby9pbmplY3QSBxoFZmFsc2UKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCjEKBE5BTUUSKRonZGFnc3Rlci1pbmdyZXNzZ2F0ZXdheS01OThiYzU5OGQ5LXA3enc0ChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KXwoFT1dORVISVhpUa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9kYWdzdGVyLWluZ3Jlc3NnYXRld2F5ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAopCg1XT1JLTE9BRF9OQU1FEhgaFmRhZ3N0ZXItaW5ncmVzc2dhdGV3YXk=’
‘x-envoy-peer-metadata-id’, ‘router~100.112.75.76~dagster-ingressgateway-598bc598d9-p7zw4.istio-system~istio-system.svc.cluster.local’
‘x-envoy-attempt-count’, ‘1’
‘x-b3-traceid’, ‘6884eec06668989ccb5ed70c16ef6f58’
‘x-b3-spanid’, ‘cb5ed70c16ef6f58’
‘x-b3-sampled’, ‘0’
‘x-envoy-original-path’, ‘/graphql’
2022-03-23T15:25:47.203134Z debug   envoy pool  queueing stream due to no available connections
2022-03-23T15:25:47.203139Z debug   envoy pool  trying to create new connection
2022-03-23T15:25:47.203144Z debug   envoy pool  creating a new connection
2022-03-23T15:25:47.203205Z debug   envoy client    [C8275] connecting
2022-03-23T15:25:47.203214Z debug   envoy connection    [C8275] connecting to 100.112.29.212:3000
2022-03-23T15:25:47.203322Z debug   envoy connection    [C8275] connection in progress
2022-03-23T15:25:47.203366Z debug   envoy http  [C8274][S2813519150144299458] request end stream
2022-03-23T15:25:47.204011Z debug   envoy connection    [C8275] connected
2022-03-23T15:25:47.205067Z debug   envoy client    [C8275] connected
2022-03-23T15:25:47.205080Z debug   envoy pool  [C8275] attaching to next stream
2022-03-23T15:25:47.205085Z debug   envoy pool  [C8275] creating stream
2022-03-23T15:25:47.205093Z debug   envoy router    [C8274][S2813519150144299458] pool ready
2022-03-23T15:25:47.205927Z debug   envoy connection    [C8275] remote close
2022-03-23T15:25:47.205937Z debug   envoy connection    [C8275] closing socket: 0
2022-03-23T15:25:47.205959Z debug   envoy connection    [C8275] SSL shutdown: rc=1
2022-03-23T15:25:47.205999Z debug   envoy client    [C8275] disconnect. resetting 1 pending requests
2022-03-23T15:25:47.206006Z debug   envoy client    [C8275] request reset
2022-03-23T15:25:47.206013Z debug   envoy router    [C8274][S2813519150144299458] upstream reset: reset reason: connection termination, transport failure reason:
2022-03-23T15:25:47.206043Z debug   envoy http  [C8274][S2813519150144299458] Sending local reply with details upstream_reset_before_response_started{connection termination}
2022-03-23T15:25:47.206079Z debug   envoy http  [C8274][S2813519150144299458] encoding headers via codec (end_stream=false):
‘:status’, ‘503’
‘content-length’, ‘95’
‘content-type’, ‘text/plain’
‘date’, ‘Wed, 23 Mar 2022 15:25:47 GMT’
‘server’, ‘istio-envoy’
2022-03-23T15:25:47.206202Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2022-03-23T15:25:47.206213Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2022-03-23T15:25:47.206217Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2022-03-23T15:25:47.206221Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2022-03-23T15:25:47.206232Z debug   envoy pool  [C8275] client disconnected, failure reason:
2022-03-23T15:25:47.206253Z debug   envoy pool  [C8275] destroying stream: 0 remaining

import logging

from mypackage import function1, function2, function3, function4, function5


def main():
    try:
        function1()
        function2()
    except Exception as e:
        logging.exception(e)
        function4()
    else:
        function5()

if __name__ == '__main__:
    main()

import logging

from dagster import success_hook, failure_hook
from mypackage import function1, function2, function3, function4, function5


@solid
def dag_function1() -> bool:
    myvar1 = True
    function1()
    return myvar1


@solid
def dag_function2() -> bool:
    myvar2 = True
    function2()
    return myvar2


@solid
def dag_function3() -> bool:
    myvar3 = True
    function3()
    return myvar3


@failure_hook
def dag_function5():
    logging.exception('NOT SURE HOW TO ACCESS MY EXCEPTION')
    function5()


@success_hook
def dag_function4():
    function4()


def main():
    dag_function3(dag_function1(), dag_function2())

dagster.core.errors.PartitionExecutionError: Error occurred during the evaluation of the `run_config_for_partition` function for partition set download_firebase_data_local_partition_set 
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/site-packages/dagster/grpc/impl.py", line 292, in get_partition_config
    return ExternalPartitionConfigData(name=partition.name, run_config=run_config)
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/site-packages/dagster/core/errors.py", line 192, in user_code_error_boundary
    raise error_cls(
The above exception was caused by the following exception:

TypeError: daily_download_config() takes 1 positional argument but 2 were given 
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/site-packages/dagster/core/errors.py", line 185, in user_code_error_boundary
    yield
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/site-packages/dagster/grpc/impl.py", line 291, in get_partition_config
    run_config = partition_set_def.run_config_for_partition(partition)
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/site-packages/dagster/core/definitions/partition.py", line 441, in run_config_for_partition
    return copy.deepcopy(self._user_defined_run_config_fn_for_partition(partition))
  File "/Users/bryan/miniconda3/envs/dagster-injector/lib/python3.9/site-packages/dagster/core/definitions/time_window_partitions.py", line 192, in <lambda>
    run_config_for_partition_fn=lambda partition: fn(

@graph
def download():
    """
    Download data from BigQuery then upload to S3
    """
    extract_data_in_date()


@daily_partitioned_config(start_date=datetime(2021, 12, 1))
def daily_download_config(date: datetime):
    return {
        "resources": {
            "date": date.strftime("%Y-%m-%d")
        }
    }

download_local_job = download.to_job(
    name=f'{NAME}_local',
    resource_defs={
        **{
            "date": make_values_resource(date=str),
            "project_name": ResourceDefinition.hardcoded_resource("test-123")
        },
        **RESOURCES_LOCAL,
    },
    config=daily_download_config,
    executor_def=in_process_executor
)

poetry run worker.py

me@LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/composite_key/compositekey$ poetry run worker.py 

  FileNotFoundError

  [Errno 2] No such file or directory: b'/snap/bin/worker.py'

  at /usr/lib/python3.8/os.py:601 in _execvpe
       597│         path_list = map(fsencode, path_list)
       598│     for dir in path_list:
       599│         fullname = path.join(dir, file)
       600│         try:
    →  601│             exec_func(fullname, *argrest)
       602│         except (FileNotFoundError, NotADirectoryError) as e:
       603│             last_exc = e
       604│         except OSError as e:
       605│             last_exc = e
me@LAPTOP-G1DAPU88:~/.ssh/workers-python/workers/composite_key/compositekey$ ls
 Citizenship.csv                   __pycache__                 dagster.yaml   pytest.ini                  simulate_alien_dict.py   tasks.py
"Gordian Algorithms' Times.xlsx"   config.yaml                 data           run_pipeline.yaml           simulate_data.ipynb      tests
 __init__.py                       currency_symbols_map.json   modules        simulate_alien_dict.ipynb   simulate_data.py         worker.py

dagster job execute -f hello_cereal.py

2021-10-22 15:33:22 - dagster - ERROR - hello_cereal_job - 272b37fb-9f39-44dc-b63a-dcd3dfbb7956 - 880 - RUN_FAILURE - Execution of run for "hello_cereal_job" failed. Pipeline failure during initialization for pipeline "hello_cereal_job". This may be due to a failure in initializing the executor or one of the loggers.

dagster.core.errors.DagsterUnmetExecutorRequirementsError: You have attempted to use an executor that uses multiple processes with an ephemeral DagsterInstance. A non-ephemeral instance is needed to coordinate execution between multiple processes. You can configure your default instance via $DAGSTER_HOME or ensure a valid one is passed when invoking the python APIs. You can learn more about setting up a persistent DagsterInstance from the DagsterInstance docs here: https://docs.dagster.io/deployment/dagster-instance#default-local-behavior

login aws
    |
    |
    |
    v
[docker_elastic, docker_kafka, sleep_10]
                                  |
                                  |
                                  |
                                  v
[node_service_one, node_service_two, node_service__three]

from dagster import resource, solid, ModeDefinition, pipeline
from dagster_aws.s3 import s3_resource

class HelperAwsS3:
    def __init__(self, s3_resource):
        self.s3_resource = s3_resource

    def s3_list_bucket(self, bucket, prefix):
        return self.s3_resource.list_objects_v2(
            Bucket=bucket,
            Prefix=prefix
        )

    def s3_download_file(self, bucket, file, local_path):
        self.s3_resource.meta.client.download_file(
            Bucket=bucket,
            Key=file,
            Filename=local_path
        )

    def s3_upload_file(self, bucket, file, local_path):
        self.s3_resource.meta.client.upload_file(
            Bucket=bucket,
            Key=file,
            Filename=local_path
        )

@resource
def connection_helper_aws_s3_resource(context):
    return HelperAwsS3()

import pathlib

import pytest
# from dagster import execute_solid
# from ontology_tagger.ontology_tagger_worker import batch_predict
# from pwmf.pipeline.utils import local_mode
import yaml
# from pycm import ConfusionMatrix
import pandas as pd
# import datetime

from ontology_tagger.modules.s3_util import S3Util

import os
import logging

cwd = pathlib.Path(__file__).parent.absolute()
s3_util = S3Util()


@pytest.fixture
def models_with_unit_test_data():
    return models('data_local')


@pytest.fixture
def models_with_functional_test_data():
    return models('functional_test')


def models(test_file):
    try:
        with open(cwd.parent / 'models.yaml') as fh:
            models = yaml.safe_load(fh)
            model_test_data = {}
            for model, meta in models.items():
                if meta['test']['skip_unit_test']:
                    continue
                if test_file in meta['test']:
                    model_test_data[model] = meta['test'][test_file]
    except Exception as e:
        print(f'Cannot load models.yaml {e}')
    return model_test_data


@pytest.fixture
def s3_paths():
    try:
        with open(cwd.parent / 'models.yaml') as fh:
            models = yaml.safe_load(fh)
            model_validation_results = {}
            for model, meta in models.items():
                path = meta['local']['model_s3'].split('.')[0]
                model_validation_results[model] = path
    except Exception as e:
        print(f'Cannot load models.yaml {e}')
    return model_validation_results


def get_config(test_file='../data/test.csv', remove_ids=False):

    if remove_ids:
        df = pd.read_csv(test_file, header=None, sep='\t', index_col=0)
        test_data = (cwd / 'temp.csv').as_posix()
        print(f'Removing ids and generating test file in {test_data}')
        df.to_csv(test_data, header=False, index=False, sep='\t')
    else:
        test_data = (cwd / test_file).as_posix()

    return {
        'resources': {
            'fqdn_string': {'config': {'fqdn': None}},
            'mds_store': {
                'config': {
                    'mds_uri': 'mongodb://root:broot@localhost:27017/mds?authSource=admin'
                }
            },
            'file_cache': {'config': {'target_folder': './'}},
            'file_manager': {'config': {}},
            's3': {},
        },
        'solids': {
            'batch_predict': {
                'inputs': {
                    'batch_size': {'value': 20},
                    'data_uri': {'value': test_data},
                    's3_uri': {'value': 'dbpedia_comprehensive'},
                    'instance_type': {'value': 'local'},
                    'batch_id': {'value': ''},
                }
            }
        },
    }


'''
@pytest.mark.unit
def test_batch_predict_local(models_with_unit_test_data):
    for model, test_data_path in models_with_unit_test_data.items():
        config = get_config('../' + test_data_path)
        config['solids']['batch_predict']['inputs']['s3_uri']['value'] = model
        dataset = config['solids']['batch_predict']['inputs']['data_uri']['value']
        print(f'testing model {model} on data {dataset}')
        output = execute_solid(batch_predict, run_config=config, mode_def=local_mode)
        assert output.success is True
        assert 'predictions' in output.output_values
        assert 'id' in output.output_values['predictions']
        assert 'probabilities' in output.output_values['predictions']
        assert 'classes' in output.output_values['predictions']

        cl = output.output_values['predictions']['classes']
        lb = output.output_values['predictions']['id']

        n_cl = len(cl[0])
        n_lb = len(lb[0])
        assert n_cl == n_lb
        for i in range(n_cl):
            accuracy = sum(1 for x, y in zip(cl, lb) if x[i] == y[i]) / float(len(cl))
            assert accuracy > 0.70


@pytest.mark.unit
def test_generate_performance_metrics(models_with_functional_test_data, s3_paths):
    """
    save metric files to s3 under the same s3 path as the model names in models.yaml
    """
    timestamp_folder = datetime.datetime.now().strftime('%Y-%m-%d_%H-%M-%S')
    for model, test_data_path in models_with_functional_test_data.items():
        config = get_config(test_data_path, remove_ids=True)
        config['solids']['batch_predict']['inputs']['s3_uri']['value'] = model
        dataset = config['solids']['batch_predict']['inputs']['data_uri']['value']
        print(f'testing model {model} on data {dataset}')
        output = execute_solid(batch_predict, run_config=config, mode_def=local_mode)
        cl = output.output_values['predictions']['classes']
        lb = output.output_values['predictions']['id']
        s3_path = s3_paths[model]
        n = len(cl[0])
        classes = []
        labels = []
        for i in range(n):
            for c, l in zip(cl, lb):
                try:
                    c_t = c[i]
                    l_t = l[i]
                    append = True
                except IndexError:
                    print(f'skipping {c} and {l} for class {i}')
                    append = False
                if append:
                    classes.append(c_t)
                    labels.append(l_t)
            cm = ConfusionMatrix(actual_vector=labels, predict_vector=classes)
            out_file = f'confusion_matrix_class{i+1}'
            print(f'Uploading metrics for file {out_file} to {s3_path}')
            cm.save_html(out_file)
            s3_util.upload_file(
                f'{out_file}.html', f'{s3_path}/{timestamp_folder}/{out_file}.html'
            )
            cm.save_csv(out_file)
            s3_util.upload_file(
                f'{out_file}.csv', f'{s3_path}/{timestamp_folder}/{out_file}.csv'
            )
'''


@pytest.mark.unit
def test_validation_classifier_dataset(models_with_unit_test_data):  # might do both ClassifierDataset and LabelMapper
    for model, test_data_path in models_with_unit_test_data.items():
        config = get_config('../' + test_data_path)
        config['solids']['batch_predict']['inputs']['s3_uri']['value'] = model
        dataset = config['solids']['batch_predict']['inputs']['data_uri']['value']
        print(f'testing model {model} on data {dataset}')

        validation_log_init()

        # ...

        num_lines = validation_log_init()
        assert num_lines == 2  # no. lines in log file expected


"""
@pytest.mark.unit
def test_validation_label_mapper(models_with_unit_test_data):
    for model, test_data_path in models_with_unit_test_data.items():
        config = get_config('../' + test_data_path)
        config['solids']['batch_predict']['inputs']['s3_uri']['value'] = model
        dataset = config['solids']['batch_predict']['inputs']['data_uri']['value']
        print(f'testing model {model} on data {dataset}')

        validation_log_init()

        # ...

        num_lines = validation_log_init()
        assert num_lines == 1  # no. lines in log file expected
"""


@pytest.fixture
def validation_log_init():
    global log_file
    log_file = 'test_ontology_tagger.log'
    open(log_file, 'w').close()  # empties file
    logging.basicConfig(filename=log_file, level=logging.INFO)


@pytest.fixture
def validation_log_check_clear():
    size = os.path.getsize(log_file)
    os.remove(log_file)  # 'global log_file'
    return size

#17 5.630             print(f'testing model {model} on data {dataset}')
#17 5.630     
#17 5.630 >           validation_log_init()
#17 5.630 
#17 5.630 tests/test_ontology_tagger.py:174: 
#17 5.631 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#17 5.631 
#17 5.631     @pytest.fixture
#17 5.631     def validation_log_init():
#17 5.631         global log_file
#17 5.631         log_file = '../notebooks/test_ontology_tagger.log'
#17 5.631 >       open(log_file, 'w').close()  # empties file
#17 5.631 E       FileNotFoundError: [Errno 2] No such file or directory: '../notebooks/test_ontology_tagger.log'
#17 5.631 
#17 5.631 tests/test_ontology_tagger.py:204: FileNotFoundError
#17 5.632 =============================== warnings summary ===============================
#17 5.632 ontology_tagger/tests/test_ontology_tagger.py::test_validation_classifier_dataset
#17 5.632   /home/worker/python/ontology_tagger/.venv/lib/python3.7/site-packages/_pytest/python.py:166: RemovedInPytest4Warning: Fixture "validation_log_init" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters. See https://docs.pytest.org/en/latest/fixture.html for more information.
#17 5.632     testfunction(**testargs)
#17 5.632 
#17 5.632 -- Docs: https://docs.pytest.org/en/latest/warnings.html
#17 5.633 ===================== 1 failed, 1 warnings in 2.36 seconds =====================
#17 ERROR: executor failed running [/bin/sh -c cd ontology_tagger && poetry run invoke deploy]: exit code: 1
------
 > [test 5/5] RUN cd ontology_tagger && poetry run invoke deploy:
------
executor failed running [/bin/sh -c cd ontology_tagger && poetry run invoke deploy]: exit code: 1
##[error]Bash exited with code '1'.
Finishing: Test worker