[Tutor] airflow dag

[Peter Otten]

shubham goyal wrote:

> He guys,
> 
> I want to ask that can we pass the parameters as commandline arguments in
> airflow when we are triggering the dag and access them inside the dag's
> python script/file.
> script:
 
> like this here i am trying to create a cluster but i need to pass password
> as cli arguments when i trigger the dag. can we do that. please help.

Have a look at argparse:

<https://docs.python.org/3.6/library/argparse.html>

Provided

(1) you are OK with the password being echoed when you type it
(2) the password doesn't require escaping to be a valid part of the
    `endpoint` argument

the code might look like this:

> from airflow import DAG
> from datetime import datetime,timedelta

  import argparse

  parser = argparse.ArgumentParser()
  parser.add_argument("passwd")
  args = parser.parse_args()

> default_args = {
>     'owner': 'airflow',
>     'depends_on_past': False,
>     'start_date': datetime.now(),
>     'email': ['airflow at airflow.com'],
>     'email_on_failure': False,
>     'email_on_retry': False
> }
> MAIN_DAG='check_dag'
> dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
> schedule_interval=None)
> 
> with open(file, "r") as f:
>     payload = f.read()  # Reading the json data from a file
>     SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
>         task_id='cluster_create',
>         method='POST',
>         http_conn_id='qubole_default',
>         # for directing to https://qa.qubole.net/api

          endpoint='/v2/clusters?auth_token=%s' % args.passwd,

>         data=payload,
>         headers={"Content-Type": "application/json"},

          params={'auth_token': args.passwd},

>         response_check=lambda response: True if response.status_code ==
>         200
> else False,
>         dag=dag
>     )
> 

To address (1) there is <https://docs.python.org/3.6/library/getpass.html>
(2) can be solved with 
<https://docs.python.org/3.6/library/urllib.parse.html>