.net - Is there a way to get distinct PartionKeys from a Table

Question

Currently I'm using the PartitionKey to differentiate devices that are storing data into Azure Table Services. I'd like to build a viewer that allows me to browse that data, but it would be nice to be able to structure it so I can view data "by device", or by PartitionKey. The viewer app won't have any knowledge of what devices exist, so it would be great if I could somehow get back a list of distinct PartionKeys in a given Table. Is this possible, or am I going to be relegated to creating a meta-data table into which I insert a new row for each device, then use that for querying?

Answer 1

Create a single table to store your partitions. Partition the table by the table names you use and add an entry for each partition you create.

public class PartitionEntry : TableServiceEntity { }

tableServiceContext.AddObject("TablePartitions", new PartitionEntry
{
    PartitionKey = "<table name>",
    RowKey = "<partition key>",
});
tableServiceContext.BeginSaveChanges(SaveChangesOptions.ContinueOnError, null, null);

then just query this table to get a list of partitions. This is very manageable to me.

var tbl = tableServiceContext.CreateQuery<PartitionEntry>("TablePartitions");
return tbl.Where(i => i.PartitionKey == "<table name>")
          .Select(i => new { PartitionKey = i.RowKey, });

I bet this could be optimized.

Answer 2

I don't think there is a way to retrieve all the partition keys. Here's a clever workaround, though: https://docs.microsoft.com/en-gb/archive/blogs/avkashchauhan/retrieving-partition-key-range-in-windows-azure-table-storage

To quote from Avkash's blog:

Digging further, I found there is no built in API to get a list of partition keys, instead I would have to create a solution for myself. So I end up inserting a single dummy row into each partition and when I wanted to get a list of partition keys I just query for those dummy items and they gave me the list I was looking for.

I'm certain you will already have seen this, but for others who may happen on this question, I think this is the best guide to table service functionality: http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-tables/ with examples and links to the detailed API docs.

Answer 3

Regretfully, Azure Tables don't have functions like distinct or others - consider it a structured key-based storage like a Dictionary in memory. Any operation you perform, will have to iterate through all the items in order to get a subset of them unless know which keys you want to load first and process that sub-list.

I would, personally, simply use a second azure table and store the partition keys there (as row keys) which then gives you an opportunity to group these by another factor. Or simply use a single partition key for this second table.

This would give you the best performance and the least amount of headache.

Sometimes, the simplest approach is the best one as you can just get the job done.

Hope this helps,

Answer 4

i tried similar approach before with:

TableQuery queryRows = new TableQuery() { SelectColumns = new List<string> { "PartitionKey" } };
... 
var tableClientSrc = storageAcctScr.CreateCloudTableClient();
var tablesSrc = tableClientSrc.ListTables();
var tableSrc = tablesSrc.FirstOrDefault(o => o.Name.Equals(nameSrc));
int cntSrc = tableSrc.ExecuteQuery(queryRows).Count();
...

upper as well as yours work very slow on large (run for 70 mln rows table - about 2 hours) or medium but with many properties table

Answer 5

This will get you a list of all the partition keys in your table:

ConcurrentDictionary<string, byte> partitionKeys = new ConcurrentDictionary<string, byte>();
Parallel.ForEach(myTable.ExecuteQuery(new TableQuery()), entity =>
{
    partitionKeys.TryAdd(entity.PartitionKey, 0);
});

Even if you have a large table, it should populate quickly becauwse it is running in parallel. There is no "ConcurrentSet," if you will, so we have to use ConcurrentDictionary. The byte is just a placeholder; all the values will be in partitionKeys.Keys.