Summary

When querying the cluster, requests could not be completed with nodes unavailable to service the query.

Symptoms

Data Manipulation Language (DML) queries (e.g. SELECT, INSERT, UPDATE) against the database fail despite all nodes being operational.

Nodes can be confirmed to be operational since Data Definition Language (DDL) queries (e.g. DESCRIBE, ALTER) return the expected results.

Below is a sample output from a cqlsh query:

cqlsh> SELECT * FROM music.albums ; 
Unable to complete request: one or more nodes were unavailable.

Cause

In this instance, the issue is a result of incorrect replication settings on the keyspace.

The sample nodetool status output shows that there is a single-datacentre named Cassandra:

Datacenter: Cassandra 
===================== 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address  Load Tokens Owns  Host ID                              Rack
UN 10.0.1.2 1.67 GB 256 36.8% 109dc707-1df2-4f50-9ec5-d611f1772f5a RAC1
UN 10.0.1.3 1.66 GB 256 30.0% a88a9570-9ead-4cdd-a81b-daf68013f8ab RAC1
UN 10.0.1.4 1.66 GB 256 33.2% 239d8975-1783-4935-addb-691173214045 RAC1

However, the music keyspace in the example above shows that the data is replicated to a non-existent datacentre named DC1:

cqlsh> DESCRIBE KEYSPACE music ;

CREATE KEYSPACE music WITH replication = { 
    'class': 'NetworkTopologyStrategy', 
    'DC1': '3' 
};
...

Solution

Update the replication settings on the keyspace with the correct datacentre name.

Getting Cassandra datacenter name in cqlsh

cqlsh> use system;
cqlsh:system> select data_center from local;

data_center
-------------
datacenter1

 

Step 1 - Alter the keyspace definition:

cqlsh> ALTER KEYSPACE music WITH replication = { 
    'class': 'NetworkTopologyStrategy', 
    'Cassandra': '3' 
};

Step 2 - (OPTIONAL) It may also be necessary to run a rolling repair around the ring:

$ nodetool repair -pr -- music

See also

KB article - Read and login failures after updating keyspace replication