python - 自动区域分类

Question

我正在从几家新闻机构抓取报道，我想创建一个过滤器来自动对报道进行分类。

现在，我的数据库中有一个包含世界所有国家/地区的表和一个包含其城市的相关表。

所以我现在很困惑如何解决这个问题。我能想到的唯一解决方案是把故事词拆分，然后将每个词与每个国家和城市进行比较，我认为这会消耗大量资源。

请指教。

Answer 1

我不确定在找到一个城市之前搜索故事是否会像世界新闻文章看起来那样非常低效。我刚看了BBC世界新闻主页，第一段提到了大部分城市。如果我声称对此有任何见解，我可能会让统计学家哭泣，但我有一种预感，这对于大多数世界新闻景点来说可能都是正确的。

不幸的是，我不精通 Python，但这里有一个 C# 程序可以实现这一点。

class Program
{
    //You could have 2 Hashsets, one for cities and one for countries and find both but you can always derive a country from a city so you won't need as much memory to reduce the load factor of the hash table. 
    //However this does mean if an article mentions only a country and not a city you can't categorize it.
    static HashSet<String> cities = new HashSet<String>();
    static Dictionary<String, String> cityToCountry = new Dictionary<String, String>();
    const int MAX_WORDS_TO_READ = 200;

    static void Main(string[] args)
    {
        addCities();
        String sentance = "Former South African President Nelson Mandela is discharged from hospital in Johannesburg after 10 days of treatment for pneumonia.";

        String city = findCity(sentance);
        String country = cityToCountry[city];

        Console.WriteLine(city + ", " + country);
        Console.ReadLine();


    }

    static String findCity(String sentance)
    {

        String word = "";
        int wordsSeen = 0;

        foreach (char c in sentance)
        {
            if (c != ' ')
            {
                word += c;
            }
            else
            {
                if (isCity(word))
                {
                    return word;
                }
                else
                {
                    word = "";
                }

                wordsSeen++;
            }

            //If you assume that if the city is not in the first n words then the article isn't about a specific city or one that you don't have in your database
            //then you can avoid having to check through massive articles that have no city
            if(wordsSeen > MAX_WORDS_TO_READ)
            {
                return null;
            }
        }

        return null;
    }

    static bool isCity(String city)
    {
        return cities.Contains(city);
    }


    static void addCities()
    {
        //Naturally you would parse in a list of cities from somewhere
        cities.Add("Berlin");
        cities.Add("London");
        cities.Add("Johannesburg");

        cityToCountry.Add("Berlin", "Germany");
        cityToCountry.Add("London", "England");
        cityToCountry.Add("Johannesburg", "South Africa");
    }




}

另一方面，如果你看一下 BBC 新闻英格兰部分，那么你最终会得到这样的故事，它甚至没有在文章本身中提到这个国家，如果你的城市列表包含奥克汉普顿，我会非常惊讶。要解决此问题，您可以使用故事所在的新闻站点部分给出的上下文，但最重要的是，此方法的效率取决于您正在抓取的新闻故事的类型。