我在为我的工作编写的程序中遇到了一堵砖墙。您不需要具体了解上下文,但长话短说,我有两个集合,每个集合大约 65 万条记录。
假设集合 A 是我知道是正确的集合,集合 B 是我知道不正确的集合。
集合 B 包含一个复杂对象,该对象具有与集合 A 中的元素相同类型的属性(换句话说,它看起来有点像这样):
// Where T : IComparable
IEnumerable<DateTime> A = ...; // Collection of T elements
IEnumerable<Complex> B = ...; // Collection of complex elements.
class Complex<DateTime>
{
public DateTime Time { get; set; }
.....
}
我的问题是我基本上需要依次枚举 A 并查看 A 的当前元素是否存在于 B 中的 Complex 对象中;如果它不存在,那么我需要创建一个 Complex 对象来封装该元素(除其他外)。
当我意识到两个列表的长度约为 650,000 个元素时,就会出现问题。我无法减少数据集;我必须使用这 650,000 个。现在我已经使用ICollection.Contains()
了,并且我尝试了二进制搜索的(天真的)实现,但这需要的时间太长了。
你对我有什么建议吗?
编辑:如果有帮助,T 实现 IComparable。EDIT2:更多上下文:使用 Linq To Objects 从 DataTable 中检索 IEnumerable。
IEnumerable<Complex> set = set.Tbl
.Where(dbObject => dbObject.TS.CompareTo(default(DateTime)) != 0)
.Select(Func<DataRow,Complex>) // Function that wraps the DataRow in a Complex object
// Just done to make debugging a little easier so we still have a large sample but small enough that it doesn't make me grow a beard
.Take(100000)
.AsEnumerable<Complex>();
为了完整起见,以防此问题被存档并且其他任何人都需要解决此问题,我当前的实现看起来有点像这样
BDataSet bSet = new BDataSet();
B_LUTableAdapter adap = new B_LUTableAdapter();
adap.Fill(bSet.B_LU);
IEnumerable<Complex> w3 = bSet.B
.Where(dbObject => dbObject.TS.CompareTo(default(DateTime)) != 0)
// Function that just wraps datarow into a complex object
.Select(Func<DataRow, Complex>)
// Just for sake of debugging speed
.Take(100000)
.AsEnumerable<Complex>();
List<Complex> b = bSet.OrderBy(x => x.Time).ToList<Complex>();
// Get last & first timestamps
// Some of the timestamps in b are 01/01/1011 for some reason,
// So we do this check.
Complex start = b.Where(x => x.Time != default(DateTime)).First();
Complex end = b.Last();
List<DateTime> a = new List<DateTime>();
// RoundSeconds reduces seconds in a DateTime to 0.
DateTime current = RoundSeconds(new DateTime(start.Time.Ticks));
while (current.CompareTo(RoundSeconds(end.Time)) <= 0)
{
a.Add(current);
current = current.Add(TimeSpan.FromMinutes(1));
}
IEnumerable<DateTime> times = b.Select(x => x.Time);
var missing = a.Where(dt => times.Contains(dt));
foreach (var dt in missing)
{
adap.Insert(dt, 0, "", "", "", null, 0, 0);
// This has since been changed to List.Add()
}
感谢 Cosmin,这个问题现在得到了解决,完成的实现是这样的: List expected = new List(); 当前日期时间 = RoundSeconds(new DateTime(start.Time.Ticks));
while (current.CompareTo(RoundSeconds(end.Time)) <= 0)
{
expected.Add(current);
current = current.Add(TimeSpan.FromMinutes(1));
}
Console.WriteLine("Expecting {0} intervals.", expected.Count);
var missing = b.FindAllMissing(expected, x => x.Time);
if(!missing.Any()) return;
Console.WriteLine("{0} missing intervals.", missing.Count());
foreach (var dt in missing)
{
b.Add(new Complex() { /* some values */ });
//Console.WriteLine("\t> Inserted new record at {0}", dt);
}
//.....
public static IEnumerable<Basic> FindAllMissing<Basic, Complex>(this IEnumerable<Complex> complexList,
IEnumerable<Basic> basicList,
Func<Complex, Basic> selector)
{
HashSet<Basic> inComplexList = new HashSet<Basic>();
foreach (Complex c in complexList)
inComplexList.Add(selector(c));
List<Basic> missing = new List<Basic>();
foreach (Basic basic in basicList)
if (!(inComplexList.Contains(basic)))
missing.Add(basic);
return missing;
}