Python programmers must use Boto3 when checking or configuring your AWS infrastructure programmatically. Most of the time, we’ll need to print the Boto3 Client’s output to the terminal in order to check it. Unfortunately, writing the boto3 response directly is not really clear.
The boto3 response is best handled by first converting it to a JSON string and then printing it to the console.
Printing Boto3 Client response
To see how we can output boto3 client’s response to the terminal we will initially use the describe_regions function of the boto3 EC2 client.
See the Python script below.
import boto3
client = boto3.client('ec2')
response = client.describe_regions(RegionNames=['us-east-1'])
print(response)
Python
Running the Python script will result in the long single line output below.
{'Regions': [{'Endpoint': 'ec2.us-east-1.amazonaws.com', 'RegionName': 'us-east-1', 'OptInStatus': 'opt-in-not-required'}], 'ResponseMetadata': {'RequestId': '9437271e-6132-468f-b19d-535f9d7bda09', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '9437271e-6132-468f-b19d-535f9d7bda09', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '449', 'date': 'Sat, 03 Apr 2021 08:30:15 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}
Plain text
In my screen it will look like this since it automatically wraps the output.
With the output above, it is hard to understand or search for the specific part of the response that you are looking for.
Note: The response object of the boto3 client is a dictionary (dict) type. The examples below will work even if you change response object to any dictionary object.
Converting Boto3 Response to JSON String
To easier understand the returned response of the boto3 client it is best to convert it to a JSON string before printing it. To change the response to a JSON string use the function json.dumps.
import boto3
import json
client = boto3.client('ec2')
response = client.describe_regions(RegionNames=['us-east-1'])
json_string = json.dumps(response, indent=2)
print(json_string)
Python
This will result in an output that is easier to understand format. The output when I ran the Python script above can be seen below.
{
"Regions": [
{
"Endpoint": "ec2.us-east-1.amazonaws.com",
"RegionName": "us-east-1",
"OptInStatus": "opt-in-not-required"
}
],
"ResponseMetadata": {
"RequestId": "f6a33cc5-58c7-4948-8f44-2769ede5f166",
"HTTPStatusCode": 200,
"HTTPHeaders": {
"x-amzn-requestid": "f6a33cc5-58c7-4948-8f44-2769ede5f166",
"cache-control": "no-cache, no-store",
"strict-transport-security": "max-age=31536000; includeSubDomains",
"content-type": "text/xml;charset=UTF-8",
"content-length": "449",
"date": "Mon, 29 Mar 2021 12:11:52 GMT",
"server": "AmazonEC2"
},
"RetryAttempts": 0
}
}
JSON
In my screen it will look like this.
That is a much easier to understand format.
The difference when converting the boto3 client response to JSON string before printing it are the following.
- The single quotes (
') are now double quotes ("). This is becausejson.dumpsconverts Python objects to JSON strings. - The output is now putting each key-value pair of the response dictionary in a newline. This is because of the
indentparameter injson.dumps. If you remove theindentparameter the output will be printed in a single line again.
We can also shorten the code by directly putting json.dumps inside the print function.
import boto3
import json
client = boto3.client('ec2')
response = client.describe_regions(RegionNames=['us-east-1'])
print(json.dumps(response, indent=2))
Python
Copy
The output will be the same as the one above that is easier to understand.
Note: The indent parameter gets a positive integer to know how many spaces it will indent to pretty-print the JSON string. You can increase it to indent=4 if you find indent=2 not noticeable.
Fixing TypeError: Object of type datetime is not JSON serializable
In the example below, we are trying to print the response of EC2 client’s describe_instances function in JSON format. This will result in a Object of type datetime is not JSON serializable error.
import boto3
import json
client = boto3.client('ec2')
response = client.describe_instances()
print(json.dumps(response, indent=2))
Python
Error Output
Traceback (most recent call last):
File "boto3_ec2_json_bad.py", line 8, in <module>
json_string = json.dumps(response, indent=2)
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\__init__.py", line 234, in dumps
return cls(
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 201, in encode
chunks = list(chunks)
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 431, in _iterencode
yield from _iterencode_dict(o, _current_indent_level)
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 405, in _iterencode_dict
yield from chunks
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 325, in _iterencode_list
yield from chunks
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 405, in _iterencode_dict
yield from chunks
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 325, in _iterencode_list
yield from chunks
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 405, in _iterencode_dict
yield from chunks
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 438, in _iterencode
o = _default(o)
File "c:\users\arnold\appdata\local\programs\python\python38\lib\json\encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type datetime is not JSON serializable
Plain text
The reason for this error is that there are values in the describe_instances response that is in the type datetime, like the LaunchTime key of each EC2 Instances.
from boto3 EC2 describe_instances documentation
There are a lot of datetime objects not only in the EC2 client, but also for other services like DynamoDB, S3, Lambda and many more.
To fix the TypeError: Object of type datetime is not JSON serializable, we will just add the default=str parameter when calling json.dumps function.
Below is the corrected Python script.
import boto3
import json
client = boto3.client('ec2')
response = client.describe_instances()
print(json.dumps(response, indent=2, default=str))
Python
Output
{
"Reservations": [
{
"Groups": [],
"Instances": [
{
"AmiLaunchIndex": 0,
"ImageId": "ami-016aa01b240468f0c",
"InstanceId": "i-05b2515c1f1d70195",
"InstanceType": "t4g.nano",
"LaunchTime": "2021-03-29 11:46:29+00:00",
"Monitoring": {
"State": "disabled"
},
"Placement": {
"AvailabilityZone": "ap-southeast-1a",
"GroupName": "",
"Tenancy": "default"
},
"PrivateDnsName": "ip-172-31-42-147.ap-southeast-1.compute.internal",
"PrivateIpAddress": "172.31.42.147",
"ProductCodes": [],
"PublicDnsName": "",
"State": {
"Code": 80,
"Name": "stopped"
},
"StateTransitionReason": "User initiated (2021-03-29 11:48:01 GMT)",
"SubnetId": "subnet-3f271e76",
"VpcId": "vpc-8f3d19e8",
"Architecture": "arm64",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"AttachTime": "2021-03-29 11:34:20+00:00",
"DeleteOnTermination": true,
"Status": "attached",
"VolumeId": "vol-00938286e87553800"
}
}
],
"ClientToken": "",
"EbsOptimized": true,
"EnaSupport": true,
"Hypervisor": "xen",
"IamInstanceProfile": {
"Arn": "arn:aws:iam::758357942546:instance-profile/ec2_ssm_role",
"Id": "AIPA2X6MXHWNVRQZPJB5C"
},
"NetworkInterfaces": [
{
"Attachment": {
"AttachTime": "2021-03-29 11:34:20+00:00",
"AttachmentId": "eni-attach-09421875a6b55f2b9",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"Status": "attached"
},
"Description": "",
"Groups": [
{
"GroupName": "TestSecurityGroup",
"GroupId": "sg-0779dca17c74a411e"
}
],
"Ipv6Addresses": [],
"MacAddress": "06:f1:78:c4:ef:c0",
"NetworkInterfaceId": "eni-0cefe51088fc1536a",
"OwnerId": "758357942546",
"PrivateDnsName": "ip-172-31-42-147.ap-southeast-1.compute.internal",
"PrivateIpAddress": "172.31.42.147",
"PrivateIpAddresses": [
{
"Primary": true,
"PrivateDnsName": "ip-172-31-42-147.ap-southeast-1.compute.internal",
"PrivateIpAddress": "172.31.42.147"
}
],
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-3f271e76",
"VpcId": "vpc-8f3d19e8",
"InterfaceType": "interface"
}
],
"RootDeviceName": "/dev/xvda",
"RootDeviceType": "ebs",
"SecurityGroups": [
{
"GroupName": "OpenVPNAcces",
"GroupId": "sg-0779dca17c74a411e"
}
],
"SourceDestCheck": true,
"StateReason": {
"Code": "Client.UserInitiatedShutdown",
"Message": "Client.UserInitiatedShutdown: User initiated shutdown"
},
"Tags": [
{
"Key": "Name",
"Value": "Test-Instance"
}
],
"VirtualizationType": "hvm",
"CpuOptions": {
"CoreCount": 2,
"ThreadsPerCore": 1
},
"CapacityReservationSpecification": {
"CapacityReservationPreference": "open"
},
"HibernationOptions": {
"Configured": false
},
"MetadataOptions": {
"State": "applied",
"HttpTokens": "optional",
"HttpPutResponseHopLimit": 1,
"HttpEndpoint": "enabled"
},
"EnclaveOptions": {
"Enabled": false
},
"BootMode": "uefi"
}
],
"OwnerId": "758357942546",
"ReservationId": "r-0533aa0fa263f1372"
}
],
"ResponseMetadata": {
"RequestId": "977f1891-045c-4b73-85d7-f47a2700c0bd",
"HTTPStatusCode": 200,
"HTTPHeaders": {
"x-amzn-requestid": "977f1891-045c-4b73-85d7-f47a2700c0bd",
"cache-control": "no-cache, no-store",
"strict-transport-security": "max-age=31536000; includeSubDomains",
"content-type": "text/xml;charset=UTF-8",
"content-length": "6928",
"vary": "accept-encoding",
"date": "Mon, 29 Mar 2021 12:17:31 GMT",
"server": "AmazonEC2"
},
"RetryAttempts": 0
}
}
JSON
The value of the default parameter in json.dumps should be a function that will convert a non-JSON serializable object, like datetime, into a JSON encodable version of the object.
Since Strings are JSON serializable, the str() function will be a perfect to convert almost all object types to Strings in Python.