45

从文件中获取行数的方法之一是 PowerShell 中的此方法：

PS C:\Users\Pranav\Desktop\PS_Test_Scripts> $a=Get-Content .\sub.ps1
PS C:\Users\Pranav\Desktop\PS_Test_Scripts> $a.count
34
PS C:\Users\Pranav\Desktop\PS_Test_Scripts>

但是，当我有一个 800 MB 的大文本文件时，如何在不读取整个文件的情况下从中获取行号？

上述方法会消耗过多的 RAM，导致脚本崩溃或完成时间过长。

4

8 回答 8

33

用于Get-Content -Read $nLinesAtTime逐部分读取文件：

$nlines = 0;

# Read file by 1000 lines at a time
gc $YOURFILE -read 1000 | % { $nlines += $_.Length };
[string]::Format("{0} has {1} lines", $YOURFILE, $nlines)

这是一个简单但缓慢的脚本来验证一个小文件的工作：

gc $YOURFILE | Measure-Object -Line
于 2012-08-23T04:50:03.600 回答
31

这是我拼凑在一起的 PowerShell 脚本，它演示了几种计算文本文件中行数的不同方法，以及每种方法所需的时间和内存。结果（如下）显示了时间和内存要求的明显差异。对于我的测试，看起来最佳点是 Get-Content，使用 ReadCount 设置为 100。其他测试需要更多的时间和/或内存使用。

#$testFile = 'C:\test_small.csv' # 245 lines, 150 KB
#$testFile = 'C:\test_medium.csv' # 95,365 lines, 104 MB
$testFile = 'C:\test_large.csv' # 285,776 lines, 308 MB

# Using ArrayList just because they are faster than Powershell arrays, for some operations with large arrays.
$results = New-Object System.Collections.ArrayList

function AddResult {
param( [string] $sMethod, [string] $iCount )
    $result = New-Object -TypeName PSObject -Property @{
        "Method" = $sMethod
        "Count" = $iCount
        "Elapsed Time" = ((Get-Date) - $dtStart)
        "Memory Total" = [System.Math]::Round((GetMemoryUsage)/1mb, 1)
        "Memory Delta" = [System.Math]::Round(((GetMemoryUsage) - $dMemStart)/1mb, 1)
    }
    [void]$results.Add($result)
    Write-Output "$sMethod : $count"
    [System.GC]::Collect()
}

function GetMemoryUsage {
    # return ((Get-Process -Id $pid).PrivateMemorySize)
    return ([System.GC]::GetTotalMemory($false))
}

# Get-Content -ReadCount 1
[System.GC]::Collect()
$dMemStart = GetMemoryUsage
$dtStart = Get-Date
$count = 0
Get-Content -Path $testFile -ReadCount 1 |% { $count++ }
AddResult "Get-Content -ReadCount 1" $count

# Get-Content -ReadCount 10,100,1000,0
# Note: ReadCount = 1 returns a string.  Any other value returns an array of strings.
# Thus, the Count property only applies when ReadCount is not 1.
@(10,100,1000,0) |% {
    $dMemStart = GetMemoryUsage
    $dtStart = Get-Date
    $count = 0
    Get-Content -Path $testFile -ReadCount $_ |% { $count += $_.Count }
    AddResult "Get-Content -ReadCount $_" $count
}

# Get-Content | Measure-Object
$dMemStart = GetMemoryUsage
$dtStart = Get-Date
$count = (Get-Content -Path $testFile -ReadCount 1 | Measure-Object -line).Lines
AddResult "Get-Content -ReadCount 1 | Measure-Object" $count

# Get-Content.Count
$dMemStart = GetMemoryUsage
$dtStart = Get-Date
$count = (Get-Content -Path $testFile -ReadCount 1).Count
AddResult "Get-Content.Count" $count

# StreamReader.ReadLine
$dMemStart = GetMemoryUsage
$dtStart = Get-Date
$count = 0
# Use this constructor to avoid file access errors, like Get-Content does.
$stream = New-Object -TypeName System.IO.FileStream(
    $testFile,
    [System.IO.FileMode]::Open,
    [System.IO.FileAccess]::Read,
    [System.IO.FileShare]::ReadWrite)
if ($stream) {
    $reader = New-Object IO.StreamReader $stream
    if ($reader) {
        while(-not ($reader.EndOfStream)) { [void]$reader.ReadLine(); $count++ }
        $reader.Close()
    }
    $stream.Close()
}

AddResult "StreamReader.ReadLine" $count

$results | Select Method, Count, "Elapsed Time", "Memory Total", "Memory Delta" | ft -auto | Write-Output

以下是包含 ~95k 行、104 MB 的文本文件的结果：

Method                                    Count Elapsed Time     Memory Total Memory Delta
------                                    ----- ------------     ------------ ------------
Get-Content -ReadCount 1                  95365 00:00:11.1451841         45.8          0.2
Get-Content -ReadCount 10                 95365 00:00:02.9015023         47.3          1.7
Get-Content -ReadCount 100                95365 00:00:01.4522507         59.9         14.3
Get-Content -ReadCount 1000               95365 00:00:01.1539634         75.4         29.7
Get-Content -ReadCount 0                  95365 00:00:01.3888746          346        300.4
Get-Content -ReadCount 1 | Measure-Object 95365 00:00:08.6867159         46.2          0.6
Get-Content.Count                         95365 00:00:03.0574433        465.8        420.1
StreamReader.ReadLine                     95365 00:00:02.5740262         46.2          0.6

以下是较大文件（包含 ~285k 行，308 MB）的结果：

Method                                    Count  Elapsed Time     Memory Total Memory Delta
------                                    -----  ------------     ------------ ------------
Get-Content -ReadCount 1                  285776 00:00:36.2280995         46.3          0.8
Get-Content -ReadCount 10                 285776 00:00:06.3486006         46.3          0.7
Get-Content -ReadCount 100                285776 00:00:03.1590055         55.1          9.5
Get-Content -ReadCount 1000               285776 00:00:02.8381262         88.1         42.4
Get-Content -ReadCount 0                  285776 00:00:29.4240734        894.5        848.8
Get-Content -ReadCount 1 | Measure-Object 285776 00:00:32.7905971         46.5          0.9
Get-Content.Count                         285776 00:00:28.4504388       1219.8       1174.2
StreamReader.ReadLine                     285776 00:00:20.4495721           46          0.4
于 2015-12-13T20:21:07.060 回答
21

这是基于 Pseudothink 帖子的单行代码。

一个特定文件中的行：

"the_name_of_your_file.txt" |% {$n = $_; $c = 0; Get-Content -Path $_ -ReadCount 1000 |% { $c += $_.Count }; "$n; $c"}

当前目录中的所有文件（单独）：

Get-ChildItem "." |% {$n = $_; $c = 0; Get-Content -Path $_ -ReadCount 1000 |% { $c += $_.Count }; "$n; $c"}

解释：

"the_name_of_your_file.txt"-> 什么都不做，只为下一步提供文件名，需要双引号
|%-> 别名 ForEach-Object，迭代提供的项目（在这种情况下只有一个），接受管道内容作为输入，当前项目保存到$_
$n = $_-> $n 作为提供的文件的名称保存以供以后使用$_，实际上这可能不需要
$c = 0-> 初始化$c为计数
Get-Content -Path $_ -ReadCount 1000-> 从提供的文件中读取 1000 行（参见线程的其他答案）
|%-> foreach 添加行数实际读取到$c（将像 1000 + 1000 + 123）
"$n; $c"-> 一旦完成读取文件，打印文件名；行数
Get-ChildItem "."-> 只是向管道添加的项目比单个文件名多

于 2017-06-16T10:00:38.513 回答
12

The first thing to try is to stream Get-Content and build up the line count one at a time, rather that storing all lines in an array at once. I think that this will give proper streaming behavior - i.e. the entire file will not be in memory at once, just the current line.

$lines = 0
Get-Content .\File.txt |%{ $lines++ }

And as the other answer suggests, adding -ReadCount could speed this up.

If that doesn't work for you (too slow or too much memory) you could go directly to a StreamReader:

$count = 0
$reader = New-Object IO.StreamReader 'c:\logs\MyLog.txt'
while($reader.ReadLine() -ne $null){ $count++ }
$reader.Close()  # Don't forget to do this. Ideally put this in a try/finally block to make sure it happens.
于 2012-08-23T05:04:56.773 回答
8

这是另一个使用 .NET 的解决方案：

[Linq.Enumerable]::Count([System.IO.File]::ReadLines("FileToCount.txt"))

它不是很容易中断，但很容易记忆。

于 2018-10-02T20:55:50.773 回答
2

对于我的一些大文件 (GB+)，SWITCH在内存上更快更容易。

注意：以下时间以分钟：秒为单位。测试的文件有 14,564,836 行，每行 906 个字符长。

1:27 SWITCH
$count = 0; switch -File $filepath { default { ++$count } }

1:39 IO.StreamReader
$reader = New-Object IO.StreamReader $filepath
while($reader.ReadLine() -ne $null){ $count++ }

1:42 Linq
$count = [Linq.Enumerable]::Count([System.IO.File]::ReadLines($filepath))

1:46 Get-Content based
$filepath |% {$file_line_count = 0; Get-Content -Path $_ -ReadCount 1000 |% { $file_line_count += $_.Count }}

如果您对发现更快的任何方法或其他方法进行了优化，请分享。

于 2020-02-27T17:52:54.750 回答
1

这是我在解析 txt 文件中的空白时尝试减少内存使用的内容。话虽如此，内存使用率仍然很高，但该过程需要更少的时间来运行。

只是为了给你一些我的文件的背景，该文件有超过 200 万条记录，并且在每行的前后都有前导空白。我相信总时间是 5 分钟以上。

$testing = 'C:\Users\something\something\test3.txt'

$filecleanup =  Get-ChildItem $testing

foreach ($file in $filecleanup)
{ 
    $file1 = Get-Content $file -readcount 1000 | foreach{$_.Trim()}
    $file1 > $filecleanup
}
于 2015-08-22T14:59:38.953 回答
0

MS DOS 命令FIND： $fileName = 'C:\dirname\filename.txt'
CMD /C ('find /v /c "" "' + $fileName + '"')
可以在文档中找到 find 命令的其他变体。

于 2020-07-15T12:05:18.893 回答