API Gateway as S3 proxy

Page content

Overview of the task

Here is the official tutorial page.

Here is an overview of the tasks.

  • Create a role contains two policies.
    • AmazonAPIGatewayPushToCloudWatchLogs
    • AmazonS3ReadOnlyAccess
  • Create paths with {} braces. This part is regarded as variable. E.g. {folder}.
  • Create method request for the paths.
    • Authorization: using AWS_IAM.
    • request path: {folder}

Pre-setting up

You should make your own IAM Role which contains two policies below.

  • AmazonAPIGatewayPushToCloudWatchLogs
  • AmazonS3ReadOnlyAccess

AmazonAPIGatewayPushToCloudWatchLogs

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:GetLogEvents",
                "logs:FilterLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

AmazonS3ReadOnlyAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "*"
        }
    ]
}

In this article, I suppose ReadOnly only API. This role is used for API authorization later.

Example 1: API which returns bucket list of your accout

At API Gateway page in AWS console.

  1. Chose “REST API -> Build”
  2. Select New API.
  3. Input your API name and simple description.
  4. Endpoint Type: Regional.

Create a method under /.

  • Method: GET
  • Integration type: AWS service
  • AWS Service: Simple Storage Service (S3)
  • Execution role: ARN of previously created role and Save.

For each API (path, method) pair, we should configure

  1. Method Request
  2. Integration Request: In this case, we build a final request to S3 bucket.
  3. Method Response
  4. Ingetration Response

Here is the simplest config.

  1. Method request
    • Authorization: AWS_IAM
  2. Integration request (you configured some of them already)
    • Action type: Use path override -> Path override /
  3. Method response
    • Add response header Content-Type for 200 status.
    • Add response 400 and 500.
  4. Integration response
    • In 200 response, create header Contents-Type as mapping value integration.response.header.Content-Type.
    • Create error code. regex 4\d{2} for 400 response and 500 also.

And test it!! Click a TEST at method page. Here is a sample respose (youBucketName could be your bucket name.)

Request: /
Status: 200
Latency: 57 ms
Response Body
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Owner>
      <ID>716a7854..................</ID>
   </Owner>
   <Buckets>
      <Bucket>
         <Name>yourBucketName</Name>
         <CreationDate>2020-07-07T12:37:34.000Z</CreationDate>
      </Bucket>
   </Buckets>
</ListAllMyBucketsResult>

Example 2: Get items in the bucket

In a nut shell, if your bucket name is foobar, than the request path should be /foobar.

Create Resource

  • Resource Path: /{bucket_name}, or any name you want, like /{arbitrary}.

Create a method under /{bucket_name}.

  • Method: GET
  • Integration type: AWS service
  • AWS Service: Simple Storage Service (S3)
  • Action type: Use path override -> Path override /{bucket} <- be careful this name. It could be arbitrary.
  • Execution role: ARN of previously created role and Save.

Here is the configuration.

  1. Method request
    • Authorization: AWS_IAM
    • Request Paths: bucket_name
    • Add request header: Content-Type
  2. Integration request (you configured some of them already)
    • Action type: Use path override -> Path override /{bucket} <- be careful this name.
    • URL path parameters:
      • Name: bucket <- This should be same with your overridden path name!!
      • Mapped from: method.request.path.bucket_name <- bucket_name comes from your resource path.
    • HTTP headers
      • x-amz-acl mapped from 'authenticated-read'
      • Content-Type mapped from method.request.header.Content-Type
    • Mapping templates: When no template matches the request Content-Type header (default)
  3. Method response
    • Add response header Content-Type for 200 status.
    • Add response 400 and 500.
  4. Integration response
    • In 200 response, create header Content-Type as mapping value integration.response.header.Content-Type.
    • Create error code. regex 4\d{2} for 400 response and 500 also.

And test it!! Click a TEST at method page. Here is a sample respose (youBucketName could be your bucket name.)

Request: /yourBucketName
Status: 200
Latency: 125 ms
Response Body
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Name>yourBucketName</Name>
   <Prefix />
   <Marker />
   <MaxKeys>1000</MaxKeys>
   <IsTruncated>false</IsTruncated>
   <Contents>
      <Key>sample2.txt</Key>
      <LastModified>2020-07-07T14:06:13.000Z</LastModified>
      <ETag>"08397a6f9517bbc8c4be351a8671f941"</ETag>
      <Size>23</Size>
      <Owner>
         <ID>716a7854..................</ID>
      </Owner>
      <StorageClass>ONEZONE_IA</StorageClass>
   </Contents>
   <Contents>
      <Key>testFolder/</Key>
      <LastModified>2020-07-07T12:38:08.000Z</LastModified>
      <ETag>"e6a7a7fee872a4564bc3995da1dfcca0"</ETag>
      <Size>0</Size>
      <Owner>
         <ID>716a7854..................</ID>
      </Owner>
      <StorageClass>STANDARD</StorageClass>
   </Contents>
   <Contents>
      <Key>testFolder/sample.txt</Key>
      <LastModified>2020-07-07T12:40:08.000Z</LastModified>
      <ETag>"64d29cfdc355e3fea88c486c2455fc14"</ETag>
      <Size>23</Size>
      <Owner>
         <ID>716a7854..................</ID>
      </Owner>
      <StorageClass>ONEZONE_IA</StorageClass>
   </Contents>
</ListBucketResult>

Here is the schematic S3 structure.

yourBucketName
├── sample2.txt
└── testFolder
    └── sample.txt

Should write from here

Remarks

Example 3: Get text contents in S3

  • path: {folder}/{item}
  • method: GET
  • Method request
    • Authorization: AWS_IAM
    • request paths: folder and item
    • request header: Content-Type
  • Integration request
    • Integration type: AWS service
    • region
    • service: S3
    • method: get
    • path: override {bucket}
    • execution role: ARN of created role
    • URL path parameters:
      • Name: bucket
      • Mapped from: method.request.path.folder
    • HTTP header
      • x-amz-acl ‘authenticated-read’
      • Content-Type method.request.header.Content-Type
    • Mapping templates: When no template matches the request Content-Type header
  • Integration Response
    • default
    • mappingtemplates: Content-Type: application/json
  • Method response
    • default
    • Response body for 200 Content-Type appication/json
Request: /yourBucketName/sample2.txt
Status: 200
Latency: 173 ms
Response Body
this is a S3 test file.

Note

Custome domain part should be configured.

Note2. PUT request (draft)

  • Create method at the path in which you want to put a file.
  • Policies in the Role: AmazonS3ReadOnlyAccess -> AmazonS3FullAccess

AmazonS3FullAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "*"
        }
    ]
}
  • Method Request
    • Request Paths: item
    • HTTP Request Headers: Content-Type
  • Integration Request
    • Path override: /{bucket}/{item}
    • Execution role: the new role
    • URL Path Parameters: Name=item, mapped from method.request.path.item
    • HTTP Headers: Content-Type, mapped from method.request.header.Content-Type

Note: Get images from S3

Settings -> Binary media types -> Add MIME types, like image/png, image/jpeg, application/pdf, etc.