Export from Google BigQuery to OCI Object Storage
Here we will see how you can export from Google BigQuery into OCI Object Storage. We will leverage Google BigQuery python APIs and OCI Functions to execute a SQL query on Big Query, securely getting the API Key and saving the resultant data into Apache Parquet files.
There are other blogs where people do similar things but I couldn’t find one for OCI Object Storage, so thought I’d share the code here. There is a blog that inspired me here from Kalif Cooper, so hat tip to Kalif.
In this example I will use the public data for top terms;
The code for this example is below, its simple python that leverages OCI Functions along with the APIs from Google and Oracle. The toughest part was the alignment of the stars for the versions of components, I ended up using Python 3.7 for this to successfully work. Now that its containerized its simple to use, it could also be parameterized further, for example it only saves as parquet;
The code in the above gist has the entire definition of the function needed, all the dependencies are nicely wrapped in the requirements.txt.
To execute I can call from the command line or even invoke it from other tools like OCI Data Integration — for that example you can use the REST endpoint and use a REST task, parameterizing everything you need.
The task can be parameterized so that you can use this single task definition and execute for different queries;
This example was on exporting from Google BugQuery, but as you can see this is useful to incorporate all kinds of REST activities. OCI Functions is a great way of adding custom business logic and they are automatically exposed with REST endpoints also! On top of that as you have seen here in the OCI Data Integration service, one of the most useful tasks is the REST Task, here we can extend data integration to call all kinds of activities. Hope you found this post useful.