S3 Support

When installed with the S3 addon spock will attempt to identify S3 URI(s) (e.g. s3://<bucket-name>/<key>) and handle them automatically. The user only needs to provide an active boto3.session.Session to an S3Config object and pass it to the SpockBuilder.

Installing

Install spock with the extra s3 related dependencies.

pip install spock-config[s3]

Creating a boto3 Session

The user must provide an active boto3.session.Session object to spock in order for the library to automatically handle S3 URI(s). Configuration is highly dependent upon your current AWS setup/security. Please refer to the boto3 docs for session and credentials for help on how to correctly configure your boto3.session.Session.

For instance, let's just suppose we are going to get our tokens via SAML authorization where we already have the SAMLAssertion, RoleArn, and PrincipalArn stored as env variables:

import boto3import os
client = boto3.client('sts')
token = client.assume_role_with_saml(    RoleArn=os.environ.get("RoleArn"), PrincipalArn=os.environ.get("PrincipalArn"),    SAMLAssertion=os.environ.get("SamlString"))
credentials = token['Credentials']
session = boto3.Session(            aws_access_key_id=credentials['AccessKeyId'],            aws_secret_access_key=credentials['SecretAccessKey'],            aws_session_token=credentials['SessionToken'],            region_name=os.environ.get('AWS_REGION'))

Using the S3Config Object

As an example let's create a basic @spock decorated class, instantiate a S3Config object from spock.addons.s3 with the boto3.session.Session we created above, and pass it to the SpockBuilder.

from spock.addons.s3 import S3Configfrom spock import SpockBuilderfrom spock import spockfrom typing import List
@spockclass BasicConfig:    """Basic spock configuration for example purposes
    Attributes:        parameter: simple boolean that flags rounding        fancy_parameter: parameter that multiplies a value        fancier_parameter: parameter that gets added to product of val and fancy_parameter        most_fancy_parameter: values to apply basic algebra to
    """    parameter: bool    fancy_parameter: float    fancier_parameter: float    most_fancy_parameter: List[int]
def main():    # Create an S3Config object and pass in the boto3 session    s3_config = S3Config(        session=session    )    # Chain the generate function to the SpockBuilder call    # Pass in the S3Config object    config = SpockBuilder(        BasicConfig,         desc='S3 example',        s3_config=s3_config    ).generate()

Defining the configuration file with a S3 URI

Usually we pass a relative or absolute system path as the configuration file command line argument. Here we pass in a S3 URI instead:

python simple.py -c s3://my-bucket/path/to/file/config.yaml

With a S3Config object passed into the SpockBuilder the S3 URI will automatically be handled by spock.

Saving to a S3 URI

Similarly, we usually pass a relative or absolute system path to the SavePath special argument type or to the user_specified_path kwarg. Again, instead we give a S3 URI:

def main():    # Create an S3Config object and pass in the boto3 session    s3_config = S3Config(        session=session    )    # Chain the generate function to the SpockBuilder call    # Pass in the S3Config object    config = SpockBuilder(        BasicConfig,         desc='S3 example',        s3_config=s3_config    ).save(user_specified_path="s3://my-bucket/path/to/file/").generate()

With a S3Config object passed into the SpockBuilder the S3 URI will automatically be handled by spock.

S3Transfer ExtraArgs

If you require any other settings for uploading or downloading files from S3 the S3Config class has two extra attributes:

download_config which takes a S3DownloadConfig object from spock.addons.s3 which supports all ExtraArgs from S3Transfer.ALLOWED_DOWNLOAD_ARGS

upload_config which takes a S3UploadConfig object from spock.addons.s3 which supports all ExtraArgs from S3Transfer.ALLOWED_UPLOAD_ARGS