javascript - Unable to scrape data via request module - Fobidden

Question

I have a wierd issue, I am trying to scrap certain page using request module, but I get 403 Access Denied on doing so. But I am perfectly able to do it using the curl module for node. But people over the internet suggests that it would be more performance hungry than requests module as I need to scrap a lot of similar pages. Why am i getting 403 when using requests module?

var options = {
  url: 'http://m.snapdeal.com/product/ostriva-antiglare-screen-protector-for/226500183',
  headers: {
    'User-Agent': 'Mozilla/5.0'
  }
}
router.get('/m', function(req, res) {
    request(options,function(err,resp,data){
        if(err){
          console.log(err);
          res.end();
          return;
        }
        console.log(resp.statusCode);
        res.send(data);
    });
});

Answer 1

服务器也在寻找Accept标头。'Accept': 'text/html;q=0.9,*/*;q=0.8',所以尝试在你的headers对象中添加类似的东西。