Hello!
I’m calling on the greater community for more and on the topic of scaling the Waku network.
Context for this is mostly 51/WAKU2-RELAY-SHARDING | Vac RFC more specifically Auto-sharding.
We only have bad solutions for now.
Choosing shards based on consistent hashing cannot evenly distribute traffic on shards and it’s does not keep app topics on the same shards either. IMO this is no better than manual assignment of apps to shards at least manually you are in control of overhead/k-anonymity. It also redistribute topics to shards when increasing the total shards in use, keeping topics on the same shards would be best.
Leaving apps to choose shards could work but only with a certain level of cooperation between apps for better k-anonymity via shard sharing.
Auto-sharding would be abandoned and the Waku network would never exist.
Lastly, we could design a new system to manage app traffic automatically. It would require significant efforts to design, implement, test and may not even be possible. I tried to come up with some design but even in theory it’s seams very hard to achieve.
Leave below your best ideas, ask questions and discuss this issue.
Thanks.