Loading SOAP results into OCI Object Storage?

Let’s see how we can make a SOAP call and upload the result into OCI Object Storage. This leverages the python zeep library which gives an easy to use interface to SOAP based services. There are tonnes of SOAP based services, the example below lists country info, Oracle UCM also has a SOAP based client so similar pattern could be used to get data from UCM and pipe it into Oracle Cloud (see this post here which leverages the Oracle OCM Web Services). You’ll see plenty of people doing things like this in other platforms — unloading / backing up Postgres using pgdump and piping it into AWS S3, there’s plenty more.

The example below invokes the ListOfCountryNamesByName SOAP api, then prints it to standard output. You will need to pip install zeep to use this.

One of the cool parts of the OCI CLI is there are tonnes of really cool features. The one I love is the OCI Object Storage put command. With this command you can take data from the standard input stream (or local files) for example and create objects in OCI!

python3 script.py | oci os object put --bucket-name mybucket --file - --name data_from_soap.json

Easy right!?

Tell me it’s easy.

Gets better. The OCI CLI Object Storage put command also supports parallel upload of parts of the file with the parallel_upload_count option;

  --parallel-upload-count INTEGER RANGE
                                  The number of parallel operations to
                                  perform. Decreasing this value will make the
                                  process less resource intensive but it may
                                  take longer. Increasing this value may
                                  decrease the time taken, but the process
                                  will consume more system resources and
                                  network bandwidth. The maximum is 1000.

You can also configure the part size depending on the data you are uploading you may wish to do this;

  --part-size INTEGER             Part size (in MiB) to use when the file is
                                  split into multiple parts and then combined.
                                  Part size must be greater than 10 MiB and
                                  defaults to 128 MiB.

There is a verify checksum capability but this is only available for when local file is used, not standard input stream.

If you want to control the overwrite behavior there are properties for this also.

  --force                         If the object name already exists, overwrite
                                  the existing object without a confirmation
                                  prompt.

  --no-overwrite                  If the object name already exists, do not
                                  overwrite the existing object.

Simple stuff to do seemingly complicated things. Next up in a post I will cover the OCI CLI Docker. Another piece in the jigsaw. See how you can run above in a serverless manner. Stay tuned!