c# - Rx framework for a web crawler

Question

I found that Rx framework looks really useful for async operations but i cannt understand how can i use it for download a lot of pages.

i am trying write something like this

var en = Enumerable.Range(0,100).Select(x => WebRequest.Create("http://google.com")).Select(x => Observable.FromAsyncPattern<WebResponse>(x.BeginGetResponse, 
    x.EndGetResponse)().Subscribe(r => Console.WriteLine(r.ContentLength)) ).ToList();

Ofcourse it doesnt work. How to do it right?

Answer 1

EDIT: Modified to provide simple error handling.

Here's what you need to do:

var urls = new[]
        {
            "http://stackoverflow.com/questions/10693617/"
                    + "rx-framework-for-a-web-crawler",
            "http://stackoverflow.com/",
            "http://stackoverflow.com/users/259769/enigmativity",
        };

Func<string, IObservable<WebResponse>> create =
    url =>
        Observable.Defer(() =>
        {
            var wr = WebRequest.Create(url);
            return
                Observable
                    .FromAsyncPattern<WebResponse>(
                        wr.BeginGetResponse,
                        wr.EndGetResponse)
                    .Invoke()
                    .Catch(Observable.Return<WebResponse>(null));
        });

var query =
    from u in urls.ToObservable()
    from r in create(u)
    select new
    {
        URL = u,
        ContentLength = r == null ? -1L : r.ContentLength,
    };

ServicePointManager.DefaultConnectionLimit = 100;

query.Subscribe(x => Console.WriteLine(x));

I would be more inclined to provide better error handling than this though. I'd send out a tuple that includes the exception rather than just a null value.