Dynamic task creation in Airflow to read multiple similar named files from sftp

Problem statement : We had an Airflow pipeline which moves a file from SFTP location into an AWS S3 bucket and then copies the data into…