All of the answers so far omit a crucial detail: code in a finally block which wraps a yield return will execute if and when IDisposable.Dispose is called upon the iterator/enumerator which executed the yield return. If outside code calls GetEnumerator() on an iterator and then, calls MoveNext() until the iterator performs the yield return within a finally block, and the outside code then abandons the enumerator without calling Dispose, the code in the finally block will not run. Depending upon what the iterator was doing, it may get annihilated by the garbage collector (though without having a chance at cleaning up any outside resources) or it may end up permanently or semi-permanently rooted as a memory leak (that could happen if, for example, it attached a lambda expression to a long-lived object's event handler).
Note that while both vb and c# are very good about ensuring that foreach loops will call Dispose on enumerators, it's possible to use iterators by calling GetEnumerator() explicitly, and it's possible some code might do so without calling Dispose(). There isn't much an iterator can do about that, but anyone writing iterators needs to be aware of the possibility.