-
Notifications
You must be signed in to change notification settings - Fork 288
Closed as not planned
Labels
Description
I dumped a local DynamoDB and then wanted to restore it in the cloud. Unfortunately, this failed with
$ dynamodump -m restore -r eu-central-2 -s MyTable
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
About to delete table MyTable. Type 'yes' to continue: yes
INFO:root:MyTable table is being deleted..
INFO:root:MyTable table is being deleted..
INFO:root:MyTable table deleted!
INFO:root:Starting restore for MyTable to MyTable..
Traceback (most recent call last):
File "/home/myaccount/.local/bin/dynamodump", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/myaccount/.local/share/pipx/venvs/dynamodump/lib/python3.12/site-packages/dynamodump/dynamodump.py", line 1549, in main
do_restore(
File "/home/myaccount/.local/share/pipx/venvs/dynamodump/lib/python3.12/site-packages/dynamodump/dynamodump.py", line 905, in do_restore
original_gsi_write_capacity = gsi["ProvisionedThroughput"][
~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'ProvisionedThroughput'
Neither providing --dataOnly
nor --billingMode PAY_PER_REQUEST
could avoid this. The GSIs in the dump look like this
"GlobalSecondaryIndexes": [
{
"IndexName": "GSI2",
"KeySchema": [
{
"AttributeName": "gsi2pk",
"KeyType": "HASH"
},
{
"AttributeName": "gsi2sk",
"KeyType": "RANGE"
}
],
"Projection": {
"ProjectionType": "ALL"
},
"IndexStatus": "ACTIVE",
"IndexSizeBytes": 2163586,
"ItemCount": 1496,
"IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/MyTable/index/GSI2"
},
they do not have a "ProvisionedThroughput". So I modified the source code
diff --git a/dynamodump/dynamodump.py b/dynamodump/dynamodump.py
index bcb2f79..90f5f41 100755
--- a/dynamodump/dynamodump.py
+++ b/dynamodump/dynamodump.py
@@ -822,7 +822,7 @@ def prepare_gsi_for_restore(gsi, billing_mode):
"Projection": gsi["Projection"],
}
- if billing_mode != PAY_PER_REQUEST_BILLING_MODE:
+ if billing_mode != PAY_PER_REQUEST_BILLING_MODE and "ProvisionedThroughput" in gsi:
result["ProvisionedThroughput"] = prepare_provisioned_throughput_for_restore(
gsi["ProvisionedThroughput"]
)
@@ -900,6 +900,9 @@ def do_restore(
original_gsi_read_capacities = []
if table_global_secondary_indexes is not None:
for gsi in table_global_secondary_indexes:
+ if "ProvisionedThroughput" not in gsi:
+ continue
+
# keeps track of original gsi write capacity units. If provisioned capacity is 0, set to
# RESTORE_WRITE_CAPACITY as fallback given that 0 is not allowed for write capacities
original_gsi_write_capacity = gsi["ProvisionedThroughput"][
@@ -1065,6 +1068,9 @@ def do_restore(
if table_global_secondary_indexes is not None:
gsi_data = []
for gsi in table_global_secondary_indexes:
+ if "ProvisionedThroughput" not in gsi:
+ continue
+
wcu = gsi["ProvisionedThroughput"]["WriteCapacityUnits"]
rcu = gsi["ProvisionedThroughput"]["ReadCapacityUnits"]
original_gsi_write_capacity = original_gsi_write_capacities.pop(0)
With these modifications, I was able to restore the database, however, I still had to pass --billingMode PAY_PER_REQUEST
to avoid
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTable operation: One or more parameter values were invalid: ProvisionedThroughput is not specified for index: GSI2
Wouldn't it be possible to get use the billing mode from the source table to avoid such issues? What do you think about integrating the proposed fix?