Skip to content

KeyError: 'ProvisionedThroughput' #311

@ufleisch

Description

@ufleisch

I dumped a local DynamoDB and then wanted to restore it in the cloud. Unfortunately, this failed with

$ dynamodump -m restore -r eu-central-2 -s MyTable
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
About to delete table MyTable. Type 'yes' to continue: yes
INFO:root:MyTable table is being deleted..
INFO:root:MyTable table is being deleted..
INFO:root:MyTable table deleted!
INFO:root:Starting restore for MyTable to MyTable..
Traceback (most recent call last):
  File "/home/myaccount/.local/bin/dynamodump", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/myaccount/.local/share/pipx/venvs/dynamodump/lib/python3.12/site-packages/dynamodump/dynamodump.py", line 1549, in main
    do_restore(
  File "/home/myaccount/.local/share/pipx/venvs/dynamodump/lib/python3.12/site-packages/dynamodump/dynamodump.py", line 905, in do_restore
    original_gsi_write_capacity = gsi["ProvisionedThroughput"][
                                  ~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'ProvisionedThroughput'

Neither providing --dataOnly nor --billingMode PAY_PER_REQUEST could avoid this. The GSIs in the dump look like this

    "GlobalSecondaryIndexes": [
      {
        "IndexName": "GSI2",
        "KeySchema": [
          {
            "AttributeName": "gsi2pk",
            "KeyType": "HASH"
          },
          {
            "AttributeName": "gsi2sk",
            "KeyType": "RANGE"
          }
        ],
        "Projection": {
          "ProjectionType": "ALL"
        },
        "IndexStatus": "ACTIVE",
        "IndexSizeBytes": 2163586,
        "ItemCount": 1496,
        "IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/MyTable/index/GSI2"
      },

they do not have a "ProvisionedThroughput". So I modified the source code

diff --git a/dynamodump/dynamodump.py b/dynamodump/dynamodump.py                                                                                                            
index bcb2f79..90f5f41 100755                                                                                                                                               
--- a/dynamodump/dynamodump.py                                                                                                                                              
+++ b/dynamodump/dynamodump.py                                                                                                                                              
@@ -822,7 +822,7 @@ def prepare_gsi_for_restore(gsi, billing_mode):                                                                                                         
         "Projection": gsi["Projection"],                                                                                                                                   
     }                                                                                                                                                                      
                                                                                                                                                                            
-    if billing_mode != PAY_PER_REQUEST_BILLING_MODE:                                                                                                                       
+    if billing_mode != PAY_PER_REQUEST_BILLING_MODE and "ProvisionedThroughput" in gsi:                                                                                    
         result["ProvisionedThroughput"] = prepare_provisioned_throughput_for_restore(                                                                                      
             gsi["ProvisionedThroughput"]                                                                                                                                   
         )                                                                                                                                                                  
@@ -900,6 +900,9 @@ def do_restore(                                                                                                                                         
     original_gsi_read_capacities = []                                                                                                                                      
     if table_global_secondary_indexes is not None:                                                                                                                         
         for gsi in table_global_secondary_indexes:                                                                                                                         
+            if "ProvisionedThroughput" not in gsi:                                                                                                                         
+                continue
+
             # keeps track of original gsi write capacity units. If provisioned capacity is 0, set to
             # RESTORE_WRITE_CAPACITY as fallback given that 0 is not allowed for write capacities
             original_gsi_write_capacity = gsi["ProvisionedThroughput"][
@@ -1065,6 +1068,9 @@ def do_restore(
             if table_global_secondary_indexes is not None:
                 gsi_data = []
                 for gsi in table_global_secondary_indexes:
+                    if "ProvisionedThroughput" not in gsi:
+                        continue
+
                     wcu = gsi["ProvisionedThroughput"]["WriteCapacityUnits"]
                     rcu = gsi["ProvisionedThroughput"]["ReadCapacityUnits"]
                     original_gsi_write_capacity = original_gsi_write_capacities.pop(0)

With these modifications, I was able to restore the database, however, I still had to pass --billingMode PAY_PER_REQUEST to avoid

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTable operation: One or more parameter values were invalid: ProvisionedThroughput is not specified for index: GSI2

Wouldn't it be possible to get use the billing mode from the source table to avoid such issues? What do you think about integrating the proposed fix?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions