File From S3 Using Curl
API Gateway supports a reasonable payload size limit of 10MB. One way towork within this limit, but still offer a means of importing largedatasets to your backend, is to allow uploads through S3. This articleshows how to use AWS Lambda to expose an S3 signed URL in response to anAPI Gateway request. Effectively, this allows you to expose a mechanismallowing users to securely upload data directly to S3, triggered by theAPI Gateway.The basic flow of the import process is as follows: the user makes an API,which is served by API Gateway and backed by a Lambda function. The Lambdafunction computes a signed URL granting upload access to an S3 bucket andreturns that to API Gateway, and API Gateway forwards the signed URL backto the user.
At this point, the user can use the existing S3 API to uploadfiles larger than 10MB. Upload through S3 signed URLIn practice, implementing this idea requires several interconnected parts:an S3 bucket, a Lambda function, and the API Gateway. Let’s walk throughhow to create a working upload system using these components.First, we need an S3 bucket for storing our data.
Curl Command To Create S3 Bucket
All objects in S3 areprivate by default and only the object owner has permission to accessthese objects. However, the object owner can optionally share objects withothers by creating a pre-signed URL, using their own security credentials,to grant time-limited permission to upload or download the objects. We are going totake advantage of this feature to allow users to upload objects to anotherwise private S3 bucket.Second, we need a Lambda function that generates pre-signed URLs inresponse to user API requests. In this example, we will use the Python AWSSDK to create our Lambda function.Third, we need to expose our Lambda function through API Gateway. Thisrequires creating a basic API that proxies requests to and from Lambda. Wewill define this API using Swagger and import it to API Gateway to startserving requests.
The S3 BucketThe S3 bucket can be created via the AWS user interface, the AWS commandline utility, or through CloudFormation. The only requirement is that thebucket be set to allow read/write permission only for the AWS user thatcreated the bucket. This is the default set of permissions for any newbucket.For example, we can easily create a new S3 bucket using AWS CLI by runningthe following command.
Aws lambda create-function -region us-east-1 -function-name urlsigner -zip-file fileb:////UrlSigner.zip -handler urlsigner.lambdahandler -runtime python2.7 -role API GatewayAt this point, we have an S3 bucket, and a Lambda function that createssigned URLs for uploading to that bucket. The last step is creating theAPI Gateway frontend that calls the Lambda function. For API Gateway toinvoke a Lambda function, you must attach a role assumable by API Gatewaythat has permission to call Lambda’s InvokeFunction action.This means you must have a role capable of being assumed by API Gatewaywith the following trust relationship.