


I built an image with the new python client and tested in stage and productin. GCP support said they recently fixed a similar bug in the java client and have added the fix to the python client in v1.15.1: I'm not sure our low level rust library (basically the c++ grpcio internals) do any retrying for us. There's a notion between timeouts and deadlines: - I know the Python client configures a timeout but I wonder if it has an inappropriate deadline for this.Īs for the Rust version, grpc clients tend to have retry logic - and ABORTED signals this failed "typically due to a concurrency issue", indicating a client should retry a certain number of times. Google.api_: 504 Deadline ExceededĪccording to it's possible the call has completed successfully despite a DEADLINE_EXCEEDED error. Response = six.next(self._response_iterator)įile "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/snapshot.py", line 45, in _restart_on_unavailableįile "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 81, in next Result = database.execute_partitioned_dml(query)įile "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/database.py", line 395, in execute_partitioned_dmlįile "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/streamed.py", line 143, in _iter_įile "/usr/local/lib/python3.7/dist-packages/google/cloud/spanner_v1/streamed.py", line 116, in _consume_next Grpc._channel._MultiThreadedRendezvous: The above exception was the direct cause of the following exception:Traceback (most recent call last):įile "/app/tools/spanner/purge_ttl.py", line 76, in įile "/app/tools/spanner/purge_ttl.py", line 65, in spanner_read_data The Python version's error is different: įile "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 79, in nextįile "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 416, in _next_įile "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 706, in _next
