AWS S3 bulk file import/export

Understand how to send and receive files to and from Pledge via AWS S3 for emissions measurement use cases

Updated over a week ago

Note: This feature is only available on custom plans. Please contact support@pledge.io to inquire

Overview

The server-to-server file exchange capability is available to enterprise customers and allows the exchange of files with Pledge via Amazon S3 in order to support automated end-to-end workflows.

To set this up for your organization, please contact your customer success manager who will arrange to provision the relevant S3 credentials to get things set up for you.

Folder structure

Folder (S3 Key ending)

What's it for?

/input/{activity_type}/

This is the folder where you drop your files that you want to be processed by Pledge for emissions computations.

/processed/{activity_type}/

After Pledge has picked up your file from /input, a copy of your file will be put here at some point shortly thereafter so that you can see that the file has been processed in some way - including if it has failed validation and been rejected.

/rejected/{activity_type}/

After Pledge first picks up your file from /input, it will undergo some basic validations to check that it conforms to the general CSV/XLSX file specifications as well as any further specifications as required by the use case.

If your file fails any of these validations, a file with the reasons for rejection will be created in the /rejected folder.

/results/{activity_type}/

If your file passed the basic validations, it will then go on to be processed for emissions computations.

At the end of the computations, one or two files will be placed in the /results folder:

  • One file containing the results of the computations and

  • Optionally, another file containing any errors that were encountered along the way

Where:

  • {activity_type} is one of: freight, last_mile, ride_hailing etc.

File naming

In order for you to correlate input files with any other subsequent files that are generated, the original filename (subject to filename validation rules) will be carried through where appropriate as per the following schema:

  • /input/{activity_type}/{orig_filename}.csv

  • /processed/{activity_type}/{orig_filename}-{yymmddhhmmss}.csv

  • /rejected/{activity_type}/{orig_filename}-{yymmddhhmmss}-errors.csv

  • /results/{activity_type}/{orig_filename}-{import_id}-{yymmddhhmmss}-results.csv

  • /results/{activity_type}/{orig_filename}-{import_id}-{yymmddhhmmss}-errors.csv

Where:

  • {activity_type} is one of: freight, last_mile, ride_hailing etc.

  • {orig_filename} is the original filename as specified by you and

  • {import_id} is an ID generated by Pledge for file that have been processed for emissions computations. You can use this ID to look up the import in the Pledge App imports area

File specifications

Input

The specification for input files is dependent on the emissions use case being targeted.

Read more here:

Processed

The specification for processed files is identical to the input files and will simply be a copy with a date and time post-pended on the filename.

Rejected

When a file is rejected, a new CSV file containing the reasons for rejection will be created with the following specification. A header row will be included.

CSV Column

Data Type

Mandatory

ROW

INTEGER

Notes: A number that identifies the row in the input file where the validation error is present

No

If the validation pertains to a specific row then this will be populated.

Otherwise it will be left empty.

COLUMN

TEXT

Notes: A column name that identifies the column in the input file where the validation error is present

No

If the validation pertains to a specific column then this will be populated with the name of the column.

Otherwise it will be left empty.

ERROR

TEXT

Notes: A description of the validation error that was encountered

Yes

See example(s) here

Results

See here for the detailed calculation results file specification.

Results - Errors

For data where emissions computations have not been computed successfully for one reason or another, a new CSV file containing the errors will be created. The specification of the file is the same as the input file but with an additional column that holds the description of the error. A header row will be included. The specification of the additional column is:

CSV Column

Data Type

Mandatory

ERROR

TEXT

Notes: A description of the error that was encountered

Yes

See example(s) here

Sample files

To help you prepare for the handling of rejections and results, some sample files for some fictional scenarios have are available here:

TBD

Notifications

If you would like to be notified by e-mail when input files get rejected or have otherwise completed processing, please contact your customer success manager to set this up.

Did this answer your question?