Executing Tasks using OCI CLI in Oracle Cloud Infrastructure Data Integration

Run a task from the OCI Console
  • Data Entity — the source and target operators in dataflows can have the data entity parameterized. You can design using a CSV, run passing a CSV or a compressed CSV or many compressed CSVs (patterns of objects)! For tables you may want to modify the table name or even have the SQL Query representing the entity passed into the execution; you can have a data entity that is a reference to an existing table/object/query or a new table/object. A use case for this could be for example a generic dataflow which has a source table say in an Oracle database and a target Parquet object say in Object Storage, each of these can be parameterized and then the task used to export a table specified at runtime along with the folder/object name where the data is written. We can then take this even further and parameterize the data asset and connection representing the database and export from an ADW or ATP or any of the other data asset types!
  • Schema/Bucket — the source and target operators in dataflows can have the schema parameterized. Many applications will have a cookie cutter approach with tenants in each schema or same logic for different buckets, the schema content/entities are the same, just switching the schema allows reuse of DataFlow for many uses. The schema can be a database schema or a bucket even.
  • Connection — sometimes the connection mechanism has to be switched at runtime for different uses for different authentication/authorization. The source and target operators in dataflows can have the connection parameterized.
  • Data Asset — many customers will have cookie cutter applications with tenants in different databases for example. If you parameterize the data asset you would have to parameterize the connection also. The source and target operators in dataflows can have the data asset parameterized.
  • Filter/Join conditions as parameters allow controlling what data is to be processed. The entire filter or join can be parameterized.
  • Expression text can use parameters — this allows allow controlling how expressions are defined at runtime. This can be configured in many operators within dataflow such as filter, join, lookup, pivot, split or expression/aggregate expressions. For example you could parameterize a filter to get orders created after a value passed as a parameter (FILTER_1.ORDER_TXNS.ORDER_DATE > $LAST_LOAD_DATE). This is an incremental extract use case from a source we could parameterize by passing the date of the last extract into a dataflow and the parameter will be used in a filter condition.
Run the task and pass runtime parameters.
oci data-integration task-run create --workspace-id YOUR_WORKSPACE_OCID --application-key YOUR_APPLICATION_KEY --registry-metadata '{"aggregator-key":"YOUR_TASK_KEY"}'
Execute the create task run command

Data Entity Parameters

Example of a CSV data file parameter named INVENTORY_DATA
Example of a JSON data file parameter named INVENTORY_DATA.
oci data-integration task-run create --workspace-id YOUR_WORKSPACE_OCID --application-key YOUR_APPLICATION_KEY --registry-metadata '{"aggregator-key":"YOUR_TASK_KEY"}' --config-provider '{"bindings":{"INVENTORY_DATA":{"rootObjectValue":{"modelType":"ENRICHED_ENTITY","entity":{"key":"dataref:0a5ae9ef-5b74-447a-9d5b-49511f3b7600/a_supplierx_stage/FILE_ENTITY:supplierx_inventory2.csv","name":"supplierx_inventory2.csv","modelType":"FILE_ENTITY","resourceName":"FILE_ENTITY:supplierx_inventory2.csv"},"dataFormat":{"formatAttribute":{"modelType":"CSV_FORMAT","encoding":"UTF-8","delimiter":",","quoteCharacter":"\"","hasHeader":true,"timestampFormat":"yyyy-MM-dd HH:mm:ss.SSS","isFilePattern":false,"escapeCharacter":"\\"},"type":"CSV","compressionConfig":{"codec":"NONE"},"isFilePattern":false}}}}}'
Monitor the tasks running from within the OCI Console
The compression options are defined when selecting the entity.

Data Asset / Connection / Schema Parameters

oci data-integration task-run create --workspace-id YOUR_WORKSPACE_OCID --application-key YOUR_APPLICATION_KEY --registry-metadata '{"aggregator-key":"YOUR_TASK_KEY"}'
--config-provider '{"bindings":{
"DATA_ASSET_PARAM":{"rootObjectValue": {"modelType":"ORACLE_ADWC_DATA_ASSET","key":"your_data_asset_key", "name":"your_data_asset_name"}},
"CONNECTION_PARAM":{"rootObjectValue":{"modelType":"ORACLE_ADWC_CONNECTION","key":"your_connection_key", "name":"your_connection_name"}},
"SCHEMA_PARAM":{"rootObjectValue":{"modelType":"SCHEMA","key":"dataref:your_connection_key/your_schema_name", "name":"your_schema_name"}}}}'

Filter / Join / Literal Parameters

oci data-integration task-run create --workspace-id YOUR_WORKSPACE_OCID --application-key YOUR_APPLICATION_KEY --registry-metadata '{"aggregator-key":"YOUR_TASK_KEY"}'
--config-provider '{"bindings":{
"FILTER_PARAMETER":{"simpleValue": "FILTER_1.USER1OBJECTS_CSV.PID is not null"},
"JOIN_PARAMETER":{"simpleValue":""},
"LITERAL_PARAMETER":{"simpleValue":"777"}}}'

--

--

--

Architect at @Oracle developing cloud services for data. Connect on Twitter @i_m_dave

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to Monitor AWS Kinesis with CloudWatch

Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

Testing Docker CVE Scanners. Part 3: Test It Yourself/Conclusions

Solving Wordle with Python

Skill Building & the Job Hunt

How to Integrate Voice call API in Various Industries?

voice api

Custom SQL Task for Executing Jobs

ELK stack basics — Deploying and configuring ELK stack(“open source Splunk”):

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Allan

David Allan

Architect at @Oracle developing cloud services for data. Connect on Twitter @i_m_dave

More from Medium

Azure Data Explorer: Real-Time Analytics — Fortinet Logs

Real-time dashboarding with Azure Event Hubs and Stream Analytics

How to use secrets without relying on .config files

Run A Small/Medium Sized Wikipedia Clone on Microsoft Azure