使用 hashmap 构建的稀疏数组对于频繁读取的数据效率非常低。最有效的实现使用允许访问分段分布的单个向量的 Trie。
Trie 可以通过仅执行只读的两个数组索引来计算表中是否存在元素,以获得存储元素的有效位置,或者知道它是否不存在于底层存储中。
它还可以为稀疏数组的默认值在后备存储中提供一个默认位置,这样您就不需要对返回的索引进行任何测试,因为 Trie 保证所有可能的源索引将至少映射到默认值在后备存储中的位置(您将经常存储零、空字符串或空对象)。
存在支持快速更新 Tries 的实现,使用可选的“compact()”操作来优化多个操作结束时后备存储的大小。尝试比哈希图快得多,因为它们不需要任何复杂的哈希函数,也不需要处理读取冲突(使用哈希图,读取和写入都有冲突,这需要一个循环跳到下一个候选位置,并对它们中的每一个进行测试以比较有效源索引...)
此外,Java Hashmaps 只能在 Objects 上建立索引,并且为每个散列源索引创建一个 Integer 对象(每次读取都需要创建这个对象,而不仅仅是写入)在内存操作方面是昂贵的,因为它强调了垃圾收集器.
我真的希望 JRE 包含一个 IntegerTrieMap<Object> 作为慢速 HashMap<Integer, Object> 或 LongTrieMap<Object> 的默认实现作为更慢的 HashMap<Long, Object> 的默认实现......但这是仍然不是这样。
你可能想知道什么是 Trie?
它只是一个小的整数数组(范围小于矩阵的整个坐标范围),允许将坐标映射到向量中的整数位置。
例如,假设您想要一个仅包含几个非零值的 1024*1024 矩阵。不是将该矩阵存储在包含 1024*1024 个元素(超过 100 万个)的数组中,您可能只想将其拆分为大小为 16*16 的子范围,而您只需要 64*64 个这样的子范围。
在这种情况下,Trie 索引将仅包含 64*64 整数 (4096),并且将至少有 16*16 数据元素(包含默认零,或在稀疏矩阵中找到的最常见的子范围)。
并且用于存储值的向量将仅包含彼此相等的子范围的 1 个副本(它们中的大多数都是零,它们将由相同的子范围表示)。
因此,不要使用类似的语法matrix[i][j]
,而是使用类似的语法:
trie.values[trie.subrangePositions[(i & ~15) + (j >> 4)] +
((i & 15) << 4) + (j & 15)]
使用 trie 对象的访问方法可以更方便地处理。
这是一个示例,内置在一个注释类中(我希望它可以编译,因为它被简化了;如果有错误要纠正,请告诉我):
/**
* Implement a sparse matrix. Currently limited to a static size
* (<code>SIZE_I</code>, <code>SIZE_I</code>).
*/
public class DoubleTrie {
/* Matrix logical options */
public static final int SIZE_I = 1024;
public static final int SIZE_J = 1024;
public static final double DEFAULT_VALUE = 0.0;
/* Internal splitting options */
private static final int SUBRANGEBITS_I = 4;
private static final int SUBRANGEBITS_J = 4;
/* Internal derived splitting constants */
private static final int SUBRANGE_I =
1 << SUBRANGEBITS_I;
private static final int SUBRANGE_J =
1 << SUBRANGEBITS_J;
private static final int SUBRANGEMASK_I =
SUBRANGE_I - 1;
private static final int SUBRANGEMASK_J =
SUBRANGE_J - 1;
private static final int SUBRANGE_POSITIONS =
SUBRANGE_I * SUBRANGE_J;
/* Internal derived default values for constructors */
private static final int SUBRANGES_I =
(SIZE_I + SUBRANGE_I - 1) / SUBRANGE_I;
private static final int SUBRANGES_J =
(SIZE_J + SUBRANGE_J - 1) / SUBRANGE_J;
private static final int SUBRANGES =
SUBRANGES_I * SUBRANGES_J;
private static final int DEFAULT_POSITIONS[] =
new int[SUBRANGES](0);
private static final double DEFAULT_VALUES[] =
new double[SUBRANGE_POSITIONS](DEFAULT_VALUE);
/* Internal fast computations of the splitting subrange and offset. */
private static final int subrangeOf(
final int i, final int j) {
return (i >> SUBRANGEBITS_I) * SUBRANGE_J +
(j >> SUBRANGEBITS_J);
}
private static final int positionOffsetOf(
final int i, final int j) {
return (i & SUBRANGEMASK_I) * MAX_J +
(j & SUBRANGEMASK_J);
}
/**
* Utility missing in java.lang.System for arrays of comparable
* component types, including all native types like double here.
*/
public static final int arraycompare(
final double[] values1, final int position1,
final double[] values2, final int position2,
final int length) {
if (position1 >= 0 && position2 >= 0 && length >= 0) {
while (length-- > 0) {
double value1, value2;
if ((value1 = values1[position1 + length]) !=
(value2 = values2[position2 + length])) {
/* Note: NaN values are different from everything including
* all Nan values; they are are also neigher lower than nor
* greater than everything including NaN. Note that the two
* infinite values, as well as denormal values, are exactly
* ordered and comparable with <, <=, ==, >=, >=, !=. Note
* that in comments below, infinite is considered "defined".
*/
if (value1 < value2)
return -1; /* defined < defined. */
if (value1 > value2)
return 1; /* defined > defined. */
if (value1 == value2)
return 0; /* defined == defined. */
/* One or both are NaN. */
if (value1 == value1) /* Is not a NaN? */
return -1; /* defined < NaN. */
if (value2 == value2) /* Is not a NaN? */
return 1; /* NaN > defined. */
/* Otherwise, both are NaN: check their precise bits in
* range 0x7FF0000000000001L..0x7FFFFFFFFFFFFFFFL
* including the canonical 0x7FF8000000000000L, or in
* range 0xFFF0000000000001L..0xFFFFFFFFFFFFFFFFL.
* Needed for sort stability only (NaNs are otherwise
* unordered).
*/
long raw1, raw2;
if ((raw1 = Double.doubleToRawLongBits(value1)) !=
(raw2 = Double.doubleToRawLongBits(value2)))
return raw1 < raw2 ? -1 : 1;
/* Otherwise the NaN are strictly equal, continue. */
}
}
return 0;
}
throw new ArrayIndexOutOfBoundsException(
"The positions and length can't be negative");
}
/**
* Utility shortcut for comparing ranges in the same array.
*/
public static final int arraycompare(
final double[] values,
final int position1, final int position2,
final int length) {
return arraycompare(values, position1, values, position2, length);
}
/**
* Utility missing in java.lang.System for arrays of equalizable
* component types, including all native types like double here.
*/
public static final boolean arrayequals(
final double[] values1, final int position1,
final double[] values2, final int position2,
final int length) {
return arraycompare(values1, position1, values2, position2, length) ==
0;
}
/**
* Utility shortcut for identifying ranges in the same array.
*/
public static final boolean arrayequals(
final double[] values,
final int position1, final int position2,
final int length) {
return arrayequals(values, position1, values, position2, length);
}
/**
* Utility shortcut for copying ranges in the same array.
*/
public static final void arraycopy(
final double[] values,
final int srcPosition, final int dstPosition,
final int length) {
arraycopy(values, srcPosition, values, dstPosition, length);
}
/**
* Utility shortcut for resizing an array, preserving values at start.
*/
public static final double[] arraysetlength(
double[] values,
final int newLength) {
final int oldLength =
values.length < newLength ? values.length : newLength;
System.arraycopy(values, 0, values = new double[newLength], 0,
oldLength);
return values;
}
/* Internal instance members. */
private double values[];
private int subrangePositions[];
private bool isSharedValues;
private bool isSharedSubrangePositions;
/* Internal method. */
private final reset(
final double[] values,
final int[] subrangePositions) {
this.isSharedValues =
(this.values = values) == DEFAULT_VALUES;
this.isSharedsubrangePositions =
(this.subrangePositions = subrangePositions) ==
DEFAULT_POSITIONS;
}
/**
* Reset the matrix to fill it with the same initial value.
*
* @param initialValue The value to set in all cell positions.
*/
public reset(final double initialValue = DEFAULT_VALUE) {
reset(
(initialValue == DEFAULT_VALUE) ? DEFAULT_VALUES :
new double[SUBRANGE_POSITIONS](initialValue),
DEFAULT_POSITIONS);
}
/**
* Default constructor, using single default value.
*
* @param initialValue Alternate default value to initialize all
* positions in the matrix.
*/
public DoubleTrie(final double initialValue = DEFAULT_VALUE) {
this.reset(initialValue);
}
/**
* This is a useful preinitialized instance containing the
* DEFAULT_VALUE in all cells.
*/
public static DoubleTrie DEFAULT_INSTANCE = new DoubleTrie();
/**
* Copy constructor. Note that the source trie may be immutable
* or not; but this constructor will create a new mutable trie
* even if the new trie initially shares some storage with its
* source when that source also uses shared storage.
*/
public DoubleTrie(final DoubleTrie source) {
this.values = (this.isSharedValues =
source.isSharedValues) ?
source.values :
source.values.clone();
this.subrangePositions = (this.isSharedSubrangePositions =
source.isSharedSubrangePositions) ?
source.subrangePositions :
source.subrangePositions.clone());
}
/**
* Fast indexed getter.
*
* @param i Row of position to set in the matrix.
* @param j Column of position to set in the matrix.
* @return The value stored in matrix at that position.
*/
public double getAt(final int i, final int j) {
return values[subrangePositions[subrangeOf(i, j)] +
positionOffsetOf(i, j)];
}
/**
* Fast indexed setter.
*
* @param i Row of position to set in the sparsed matrix.
* @param j Column of position to set in the sparsed matrix.
* @param value The value to set at this position.
* @return The passed value.
* Note: this does not compact the sparsed matric after setting.
* @see compact(void)
*/
public double setAt(final int i, final int i, final double value) {
final int subrange = subrangeOf(i, j);
final int positionOffset = positionOffsetOf(i, j);
// Fast check to see if the assignment will change something.
int subrangePosition, valuePosition;
if (Double.compare(
values[valuePosition =
(subrangePosition = subrangePositions[subrange]) +
positionOffset],
value) != 0) {
/* So we'll need to perform an effective assignment in values.
* Check if the current subrange to assign is shared of not.
* Note that we also include the DEFAULT_VALUES which may be
* shared by several other (not tested) trie instances,
* including those instanciated by the copy contructor. */
if (isSharedValues) {
values = values.clone();
isSharedValues = false;
}
/* Scan all other subranges to check if the position in values
* to assign is shared by another subrange. */
for (int otherSubrange = subrangePositions.length;
--otherSubrange >= 0; ) {
if (otherSubrange != subrange)
continue; /* Ignore the target subrange. */
/* Note: the following test of range is safe with future
* interleaving of common subranges (TODO in compact()),
* even though, for now, subranges are sharing positions
* only between their common start and end position, so we
* could as well only perform the simpler test <code>
* (otherSubrangePosition == subrangePosition)</code>,
* instead of testing the two bounds of the positions
* interval of the other subrange. */
int otherSubrangePosition;
if ((otherSubrangePosition =
subrangePositions[otherSubrange]) >=
valuePosition &&
otherSubrangePosition + SUBRANGE_POSITIONS <
valuePosition) {
/* The target position is shared by some other
* subrange, we need to make it unique by cloning the
* subrange to a larger values vector, copying all the
* current subrange values at end of the new vector,
* before assigning the new value. This will require
* changing the position of the current subrange, but
* before doing that, we first need to check if the
* subrangePositions array itself is also shared
* between instances (including the DEFAULT_POSITIONS
* that should be preserved, and possible arrays
* shared by an external factory contructor whose
* source trie was declared immutable in a derived
* class). */
if (isSharedSubrangePositions) {
subrangePositions = subrangePositions.clone();
isSharedSubrangePositions = false;
}
/* TODO: no attempt is made to allocate less than a
* fully independant subrange, using possible
* interleaving: this would require scanning all
* other existing values to find a match for the
* modified subrange of values; but this could
* potentially leave positions (in the current subrange
* of values) unreferenced by any subrange, after the
* change of position for the current subrange. This
* scanning could be prohibitively long for each
* assignement, and for now it's assumed that compact()
* will be used later, after those assignements. */
values = setlengh(
values,
(subrangePositions[subrange] =
subrangePositions = values.length) +
SUBRANGE_POSITIONS);
valuePosition = subrangePositions + positionOffset;
break;
}
}
/* Now perform the effective assignment of the value. */
values[valuePosition] = value;
}
}
return value;
}
/**
* Compact the storage of common subranges.
* TODO: This is a simple implementation without interleaving, which
* would offer a better data compression. However, interleaving with its
* O(N²) complexity where N is the total length of values, should
* be attempted only after this basic compression whose complexity is
* O(n²) with n being SUBRANGE_POSITIIONS times smaller than N.
*/
public void compact() {
final int oldValuesLength = values.length;
int newValuesLength = 0;
for (int oldPosition = 0;
oldPosition < oldValuesLength;
oldPosition += SUBRANGE_POSITIONS) {
int oldPosition = positions[subrange];
bool commonSubrange = false;
/* Scan values for possible common subranges. */
for (int newPosition = newValuesLength;
(newPosition -= SUBRANGE_POSITIONS) >= 0; )
if (arrayequals(values, newPosition, oldPosition,
SUBRANGE_POSITIONS)) {
commonSubrange = true;
/* Update the subrangePositions|] with all matching
* positions from oldPosition to newPosition. There may
* be several index to change, if the trie has already
* been compacted() before, and later reassigned. */
for (subrange = subrangePositions.length;
--subrange >= 0; )
if (subrangePositions[subrange] == oldPosition)
subrangePositions[subrange] = newPosition;
break;
}
if (!commonSubrange) {
/* Move down the non-common values, if some previous
* subranges have been compressed when they were common.
*/
if (!commonSubrange && oldPosition != newValuesLength) {
arraycopy(values, oldPosition, newValuesLength,
SUBRANGE_POSITIONS);
/* Advance compressed values to preserve these new ones. */
newValuesLength += SUBRANGE_POSITIONS;
}
}
}
/* Check the number of compressed values. */
if (newValuesLength < oldValuesLength) {
values = values.arraysetlength(newValuesLength);
isSharedValues = false;
}
}
}
注意:此代码不完整,因为它处理单个矩阵大小,并且它的压缩器仅限于检测常见的子范围,而不是交错它们。
此外,代码不会根据矩阵大小确定用于将矩阵拆分为子范围(对于 x 或 y 坐标)的最佳宽度或高度。它只使用相同的静态子范围大小 16(对于两个坐标),但它可以方便地使用 2 的任何其他小幂(但非 2 的幂会减慢int indexOf(int, int)
和int offsetOf(int, int)
内部方法),独立于两个坐标和向上到矩阵的最大宽度或高度。理想情况下,该compact()
方法应该能够确定最佳拟合尺寸。
如果这些拆分子范围的大小可以变化,则需要为这些子范围大小添加实例成员而不是 static SUBRANGE_POSITIONS
,并使静态方法int subrangeOf(int i, int j)
和int positionOffsetOf(int i, int j)
非静态方法;和初始化数组DEFAULT_POSITIONS
,并且DEFAULT_VALUES
需要以不同的方式删除或重新定义。
如果您想支持交错,基本上您将首先将现有值分成两个大小相同的值(两者都是最小子范围大小的倍数,第一个子集可能比第二个子范围多一个子范围),并且您将在所有连续位置扫描较大的一个以找到匹配的交错;那么您将尝试匹配这些值。然后,您将通过将子集分成两半(也是最小子范围大小的倍数)来递归循环,然后再次扫描以匹配这些子集(这会将子集的数量乘以 2:您必须想知道是否加倍subrangePositions 索引的大小与值的现有大小相比值得该值,以查看它是否提供有效的压缩(如果没有,则停在那里:您已经直接从交错压缩过程中找到了最佳子范围大小)。在这种情况下; 在压缩期间,子范围大小将是可变的。
但是此代码显示了如何分配非零值并data
为其他(非零)子范围重新分配数组,然后如何优化(compact()
使用该方法执行分配后setAt(int i, int j, double value)
)在存在重复时存储此数据可以在数据中统一的子范围,并在subrangePositions
数组中的相同位置重新索引。
无论如何,trie 的所有原则都在那里实现:
使用单个向量而不是数组的双索引数组(每个单独分配)来表示矩阵总是更快(并且在内存中更紧凑,意味着更好的局部性)。改进在方法中是可见的double getAt(int, int)
!
您节省了大量空间,但是在分配值时可能需要一些时间来重新分配新的子范围。出于这个原因,子范围不应太小,否则重新分配将过于频繁地发生以设置矩阵。
通过检测常见的子范围,可以将初始大矩阵自动转换为更紧凑的矩阵。一个典型的实现将包含一个compact()
如上所述的方法。但是,如果 get() 访问非常快而 set() 非常快,如果有很多公共子范围要压缩(例如,用自身减去一个大的非稀疏随机填充矩阵时,compact() 可能会非常慢,或将其乘以零:在这种情况下,通过实例化一个新的树并删除旧的树来重置树会更简单、更快)。
公共子范围在数据中使用公共存储,因此该共享数据必须是只读的。如果必须更改单个值而不更改矩阵的其余部分,则必须首先确保它在subrangePositions
索引中仅被引用一次。否则,您需要在values
向量的任何位置(方便地在末尾)分配一个新的子范围,然后将这个新子范围的位置存储到subrangePositions
索引中。
请注意,通用 Colt 库虽然非常好,但在处理稀疏矩阵时却没有那么好,因为它使用散列(或行压缩)技术,尽管它是一个暂时不支持尝试的技术出色的优化,既节省空间又节省时间,尤其是对于最频繁的 getAt() 操作。
即使是这里描述的尝试的 setAt() 操作也节省了大量时间(这里实现的方式,即设置后不自动压缩,仍然可以根据需求和估计时间来实现,压缩仍然可以节省大量存储空间时间的代价):节省的时间与子范围内的单元格数量成正比,而节省的空间与每个子范围内的单元格数量成反比。如果然后使用子范围大小,例如每个子范围的单元格数是 2D 矩阵中单元格总数的平方根(使用 3D 矩阵时它将是三次方根),那么这是一个很好的折衷方案。
Colt 稀疏矩阵实现中使用的散列技术的不便之处在于它们增加了大量的存储开销,并且由于可能的冲突而导致访问时间变慢。尝试可以避免所有冲突,然后可以保证在最坏的情况下将线性 O(n) 时间节省到 O(1) 时间,其中 (n) 是可能的冲突数(在稀疏矩阵的情况下,可能是最多矩阵中非默认值单元的数量,即最多矩阵大小的总数乘以与散列填充因子成比例的因子,对于非稀疏即完整矩阵)。
Colt 中使用的 RC(行压缩)技术与 Tries 更接近,但这是另一个代价,这里使用的压缩技术,对于最频繁的只读 get() 操作的访问时间非常慢,而且非常慢setAt() 操作的压缩。此外,使用的压缩不是正交的,这与保留正交性的 Tries 演示不同。尝试也将为相关的查看操作保留这种正交性,例如跨步、转置(被视为基于整数循环模块化操作的跨步操作)、子范围(以及一般的子选择,包括排序视图)。
我只是希望 Colt 将在将来更新以使用 Tries 实现另一个实现(即 TrieSparseMatrix 而不仅仅是 HashSparseMatrix 和 RCSparseMatrix)。这些想法在这篇文章中。
Trove 实现(基于 int->int 映射)也基于类似于 Colt 的 HashedSparseMatrix 的散列技术,即它们具有相同的不便之处。尝试会快得多,消耗的额外空间适中(但这个空间可以优化,甚至在延迟时间内变得比 Trove 和 Colt 更好,对结果矩阵/trie 使用最终的 compact()ion 操作)。
注意:这个 Trie 实现绑定到特定的本机类型(这里是双精度)。这是自愿的,因为使用装箱类型的通用实现具有巨大的空间开销(并且访问时间要慢得多)。在这里,它只使用双精度的原生一维数组,而不是通用向量。但是对于 Tries 来说,当然也可以派生一个泛型实现......不幸的是,Java 仍然不允许编写具有本地类型的所有优点的真正泛型类,除非通过编写多个实现(对于泛型 Object 类型或为每个本机类型),并通过类型工厂提供所有这些操作。该语言应该能够自动实例化本机实现并自动构建工厂(现在即使在 Java 7 中也不是这种情况,这是 .