Pandas, Functions and Data Integration

  • the function itself
  • the input parameters
  • the output parameters
  • the function configuration attributes (by default there is one named BATCH_SIZE=10000 which controls the batches each function will be given), you can also add in attributes to control how the function behaves.
Viewing the upstream input data for the function
Q1,Q2,Q3,Q4 were added as inputs to function.
The var_name (quarter) and value_name (sales) attributes were added as function configuration.
Define the output attributes — quarter and sales
View the data
echo '{"data":"<base64 encoded JSON document>","parameters":{"parameter_name":"parameter_value", "parameter_name":"parameter_value", "parameter_name":"parameter_value"}}' | fn invoke <fnapplicationname> <fnname>
{"secret_id_field":1,"staff_no":9999, "name":"Dean McGrath", "2016":349, "2017":231, "2018":876, "2019":679}
echo '{"data":"eyJzZWNyZXRfaWRfZmllbGQiOjEsInN0YWZmX25vIjo5OTk5LCAibmFtZSI6IkRlYW4gTWNHcmF0aCIsICIyMDE2IjozNDksICIyMDE3IjoyMzEsICIyMDE4Ijo4NzYsICIyMDE5Ijo2Nzl9Cg==","parameters":{"id_cols":"staff_no,name","var_name":"year","value_name":"hours"}}' | fn invoke diworkshopapp unpivot

Debugging

debugparam = input_parameters.get("debug")debug = "false"if (debugparam is not None):    debug = debugparam# Example to print the input data to the OCI Functions log
if debug == "true":
print(input_data, flush=True)

Common Issues

  1. Check your policies — ensure OCI Data Integration can invoke your service. Also check that your function can use the dependencies you have.
  2. If you see blank values or get java.lang.NumberFormatException check that the attributes you defined on the output match the response from your function — for example check case/spellings.
  3. when viewing data in your dataflow and you see “com.oracle.bmc.model.BmcException: (-1, null, false) Processing exception while communicating to: https://functions.null.oci.oraclecloud.com (outbound opc-request-id: nnnnnnnn)” check that you have actually selected an OCI Function in the Details panel of the Fn operator within the dataflow in OCI Data Integration.
  4. when viewing data in your dataflow and you see something like “com.oracle.bmc.model.BmcException: (502, FunctionInvokeExecutionFailed, false) function failed (opc-request-id: nnnnnn)” check that you have mapped the input attributes of the function.

Summary

--

--

--

Architect at @Oracle developing cloud services for data. Connect on Twitter @i_m_dave

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Assisting People at Haptik Using Amazon Polly | Amazon Web Services

How to automate Python code formatting

APIs and Tyres — A common pattern

Early Data Checks Saves Time — and Money

On CRM: Why A “Low Code” CRM Could Be A Game Changer For Smaller Companies

Georgia Tech OMSCS: CS6515 Graduate Algorithms

Technical Research Article on Continuous Integration

OSCA Day Abuja 2022 | The Future is Sanchez

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
David Allan

David Allan

Architect at @Oracle developing cloud services for data. Connect on Twitter @i_m_dave

More from Medium

Lambda Function in Python

Using Python to Convert IP Addresses into User Locations

The easiest way to parallelize pandase

Pandas for data analysis