This folder is also mapped to the docker containers inside the docker-compose.yaml file. Unit test for airflow dags has been defined and present in the test folder. Create multiple databases in postgres as part of docker compose.To populate child tables when master table is populated.To get the execution date in the Daily DAG.Adding task retires with Task Flow style dag definition.Using Task Flow with classic operators.Calls the NY Helath API with that date and saves result in json.Gets execution date from Airflow context.For example t1 DockerOperator ( taskid'task', dagdag, image'test-runner:1. The next day activate the Dag : LOAD_NY_COVID_DLY. Need to access schedule time in airflows docker operator.Loads this json data to Postgres table.Makes a call to NY Health API and gets data in json.Creates function and trigger in postgres to auto-populate the all the 62 counties with their data.Created tables for all 62 counties in New York in Postgres under database userdata.These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.Ĭlone this repo to your machine and follow the instructions under Usage □ UsageĪ step by step series of examples that tell you how to get a development env running.Īirflow connections add 'postgres_new' -conn-uri to postgres and create new database with name 'userdata' The task should execute at a specific time everyday and populate the tables. The objective here is to get NY Covid testing data for all counties and persist the data in a database. Two dags named NY_COVID_INITIAL_LD and LOAD_NY_COVID_DLY will help us get historical and ongoing daily loads from New York Covid testing data. Get covid testing data from NY Health API and persist in Postgres □ Table of Contents It will also include some sample DAGs and workflows. airflow-docker This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. ![]() Airflow-docker | This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |