Remember in consistent hashing, each shard has a big range of hash values to map objects. With this set-up data load may not balanced among different shards.
Instead of having one big range for each shard, what about adding marker to split the big range of hash values into smaller chunks and those smaller hash range will assign to different shards on different server. Thus helping with load balancing, but How?
Lets we have 20 objects to store and I still have 4 shards available each having different range of hash values equal in length. But may be out of 20 objects 14 are mapped to Shard 0 and rest are equally distributed to 1, 2 and 3. Unbalanced, Is not it? This is the case where smaller hash range can help a lot in load balancing.
Consistent hashing can use multiple markers for shards for mapping several smaller range of hash
values instead of one big range.
It will have two positive effects:
1. When the new shard is to be added, that shard will get more objects from all existing shards in the server, instead of just few from neighboring shards.Similarly, One of the existing shard is to be removed all objects that shard was holding will evenly distributed to all existing shards.
2. The overall distribution of data will be fairly even. - Helps with load balancing,
On my next entry, I will discuss another exciting design problem. Stay tune! See you soon!!
Instead of having one big range for each shard, what about adding marker to split the big range of hash values into smaller chunks and those smaller hash range will assign to different shards on different server. Thus helping with load balancing, but How?
Lets we have 20 objects to store and I still have 4 shards available each having different range of hash values equal in length. But may be out of 20 objects 14 are mapped to Shard 0 and rest are equally distributed to 1, 2 and 3. Unbalanced, Is not it? This is the case where smaller hash range can help a lot in load balancing.
Consistent hashing can use multiple markers for shards for mapping several smaller range of hash
values instead of one big range.
It will have two positive effects:
1. When the new shard is to be added, that shard will get more objects from all existing shards in the server, instead of just few from neighboring shards.Similarly, One of the existing shard is to be removed all objects that shard was holding will evenly distributed to all existing shards.
2. The overall distribution of data will be fairly even. - Helps with load balancing,
On my next entry, I will discuss another exciting design problem. Stay tune! See you soon!!
Comments
Post a Comment