I'm not sure a single regex would be able to suit your needs: these nested substrings always mess it up.
One solution could be the following algorithm (written in Java, but I guess the translation to C# won't be that hard):
/**
* Finds all matches (i.e. including sub/nested matches) of the regex in the input string.
*
* @param input
* The input string.
* @param regex
* The regex pattern. It has to target the most nested substrings. For example, given the following input string
* <code>A{01B{23}45C{67}89}</code>, if you want to catch every <code>X{*}</code> substrings (where <code>X</code> is a capital letter),
* you have to use <code>[A-Z][{][^{]+?[}]</code> or <code>[A-Z][{][^{}]+[}]</code> instead of <code>[A-Z][{].+?[}]</code>.
* @param format
* The format must follow the <a href= "http://docs.oracle.com/javase/7/docs/api/java/util/Formatter.html#syntax" >format string
* syntax</a>. It will be given one single integer as argument, so it has to contain (and to contain only) a <code>%d</code> flag. The
* format must not be foundable anywhere in the input string. If <code>null</code>, <code>ééé%dèèè</code> will be used.
* @return The list of all the matches of the regex in the input string.
*/
public static List<String> findAllMatches(String input, String regex, String format) {
if (format == null) {
format = "ééé%dèèè";
}
int counter = 0;
Map<String, String> matches = new LinkedHashMap<String, String>();
Pattern pattern = Pattern.compile(regex);
Matcher matcher = pattern.matcher(input);
// if a substring has been found
while (matcher.find()) {
// create a unique replacement string using the counter
String replace = String.format(format, counter++);
// store the relation "replacement string --> initial substring" in a queue
matches.put(replace, matcher.group());
String end = input.substring(matcher.end(), input.length());
String start = input.substring(0, matcher.start());
// replace the found substring by the created unique replacement string
input = start + replace + end;
// reiterate on the new input string (faking the original matcher.find() implementation)
matcher = pattern.matcher(input);
}
List<Entry<String, String>> entries = new LinkedList<Entry<String, String>>(matches.entrySet());
// for each relation "replacement string --> initial substring" of the queue
for (int i = 0; i < entries.size(); i++) {
Entry<String, String> current = entries.get(i);
// for each relation that could have been found before the current one (i.e. more nested)
for (int j = 0; j < i; j++) {
Entry<String, String> previous = entries.get(j);
// if the current initial substring contains the previous replacement string
if (current.getValue().contains(previous.getKey())) {
// replace the previous replacement string by the previous initial substring in the current initial substring
current.setValue(current.getValue().replace(previous.getKey(), previous.getValue()));
}
}
}
return new LinkedList<String>(matches.values());
}
Thus, in your case:
String input = "tes{}tR{R{abc}aD{mnoR{xyz}}}";
String regex = "[A-Z][{][^{}]+[}]";
findAllMatches(input, regex, null);
Returns:
R{abc}
R{xyz}
D{mnoR{xyz}}
R{R{abc}aD{mnoR{xyz}}}