Pandas, Functions and Data Integration

  • the function itself
  • the input parameters
  • the output parameters
  • the function configuration attributes (by default there is one named BATCH_SIZE=10000 which controls the batches each function will be given), you can also add in attributes to control how the function behaves.
Viewing the upstream input data for the function
Q1,Q2,Q3,Q4 were added as inputs to function.
The var_name (quarter) and value_name (sales) attributes were added as function configuration.
Define the output attributes — quarter and sales
View the data
echo '{"data":"<base64 encoded JSON document>","parameters":{"parameter_name":"parameter_value", "parameter_name":"parameter_value", "parameter_name":"parameter_value"}}' | fn invoke <fnapplicationname> <fnname>
{"secret_id_field":1,"staff_no":9999, "name":"Dean McGrath", "2016":349, "2017":231, "2018":876, "2019":679}
echo '{"data":"eyJzZWNyZXRfaWRfZmllbGQiOjEsInN0YWZmX25vIjo5OTk5LCAibmFtZSI6IkRlYW4gTWNHcmF0aCIsICIyMDE2IjozNDksICIyMDE3IjoyMzEsICIyMDE4Ijo4NzYsICIyMDE5Ijo2Nzl9Cg==","parameters":{"id_cols":"staff_no,name","var_name":"year","value_name":"hours"}}' | fn invoke diworkshopapp unpivot

Debugging

debugparam = input_parameters.get("debug")debug = "false"if (debugparam is not None):    debug = debugparam# Example to print the input data to the OCI Functions log
if debug == "true":
print(input_data, flush=True)

Common Issues

  1. Check your policies — ensure OCI Data Integration can invoke your service. Also check that your function can use the dependencies you have.
  2. If you see blank values or get java.lang.NumberFormatException check that the attributes you defined on the output match the response from your function — for example check case/spellings.
  3. when viewing data in your dataflow and you see “com.oracle.bmc.model.BmcException: (-1, null, false) Processing exception while communicating to: https://functions.null.oci.oraclecloud.com (outbound opc-request-id: nnnnnnnn)” check that you have actually selected an OCI Function in the Details panel of the Fn operator within the dataflow in OCI Data Integration.
  4. when viewing data in your dataflow and you see something like “com.oracle.bmc.model.BmcException: (502, FunctionInvokeExecutionFailed, false) function failed (opc-request-id: nnnnnn)” check that you have mapped the input attributes of the function.

Summary

--

--

--

Architect at @Oracle developing cloud services for data. Connect on Twitter @i_m_dave

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to root Innjoo One 3g

Root LG Phone

Recursion

Deploy Nomad and Consul Using Ansible on GCP

Introduction to HashMap in Java

Explain By Example: Event Grid

How To Transfer 3DS SD Card Data

Don’t Overwrite Tags You Run Containers With (But Do Overwrite Tags You Build Image

Image Stitching Using OpenCV — Part 1

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Allan

David Allan

Architect at @Oracle developing cloud services for data. Connect on Twitter @i_m_dave

More from Medium

How to Read Data from Hive & Write to MS SQL Table Using Spark-Shell

Apache Spark >> Create a Local Apache Spark Development Environment with Just Two Commands

How to setup PySpark on Windows?

A Beginner’s Guide to Scrape Data about Mobile Phones from Shopee and Perform ETL by using Python