13

当我通过 创建 zip 存档时java.util.zip.*,有没有办法将生成的存档拆分为多个卷?

假设我的整个存档有一个filesize24 MB我想将它分成 3 个文件,每个文件的大小限制为 10 MB。
是否有具有此功能的 zip API?或者任何其他好的方法来实现这一点?

谢谢托尔斯滕

4

5 回答 5

10

检查:http ://saloon.javaranch.com/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic&f=38&t=004618

我不知道有任何公共 API 可以帮助您做到这一点。(尽管如果您不想以编程方式执行此操作,可以使用诸如 WinSplitter 之类的实用程序来执行此操作)

我没有尝试过,但是使用 ZippedInput/OutputStream 时的每个 ZipEntry 都有一个压缩大小。在创建压缩文件时,您可能会粗略估计压缩文件的大小。如果您需要 2MB 的压缩文件,则可以在条目的累积大小变为 1.9MB 后停止写入文件,清单文件和其他 zip 文件特定元素需要 0.1MB。 因此,简而言之,您可以在 ZippedInputStream 上编写一个包装器,如下所示:

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class ChunkedZippedOutputStream {

    private ZipOutputStream zipOutputStream;

    private final String path;
    private final String name;

    private long currentSize;
    private int currentChunkIndex;
    private final long MAX_FILE_SIZE = 16000000; // Whatever size you want
    private final String PART_POSTFIX = ".part.";
    private final String FILE_EXTENSION = ".zip";

    public ChunkedZippedOutputStream(String path, String name) throws FileNotFoundException {
        this.path = path;
        this.name = name;
        constructNewStream();
    }

    public void addEntry(ZipEntry entry) throws IOException {
        long entrySize = entry.getCompressedSize();
        if ((currentSize + entrySize) > MAX_FILE_SIZE) {
            closeStream();
            constructNewStream();
        } else {
            currentSize += entrySize;
            zipOutputStream.putNextEntry(entry);
        }
    }

    private void closeStream() throws IOException {
        zipOutputStream.close();
    }

    private void constructNewStream() throws FileNotFoundException {
        zipOutputStream = new ZipOutputStream(new FileOutputStream(new File(path, constructCurrentPartName())));
        currentChunkIndex++;
        currentSize = 0;
    }

    private String constructCurrentPartName() {
        // This will give names is the form of <file_name>.part.0.zip, <file_name>.part.1.zip, etc.
        return name + PART_POSTFIX + currentChunkIndex + FILE_EXTENSION;
    }
}

上面的程序只是一种方法的提示,无论如何都不是最终的解决方案

于 2008-10-28T16:52:07.317 回答
5

如果目标是让输出与 pkzip 和 winzip 兼容,我不知道有任何开源库可以做到这一点。我们对我们的一个应用程序有类似的要求,我最终编写了自己的实现(与 zip 标准兼容)。如果我记得,对我们来说最困难的事情是我们必须即时生成单个文件(大多数 zip 实用程序的工作方式是它们创建大 zip 文件,然后再返回并拆分它 - 这更容易实现。花了大约一天的时间来编写和 2 天的时间来调试。

zip 标准解释了文件格式的外观。如果你不怕卷起袖子,这绝对是可行的。您必须自己实现一个 zip 文件生成器,但您可以使用 Java 的 Deflator 类为压缩数据生成分段流。您必须自己生成文件和节标题,但它们只是字节 - 一旦您深入研究就不会太难了。

这是zip 规范- K 部分包含您正在寻找的具体信息,但您还需要阅读 A、B、C 和 F。如果您正在处理非常大的文件(我们曾经是),您也必须使用 Zip64 的东西 - 但对于 24 MB,您很好。

如果您想潜入并尝试一下-如果遇到问题,请回帖,我会看看是否可以提供一些指示。

于 2008-10-29T02:34:13.990 回答
1

对于它的价值,我喜欢在任何地方使用try-with-resources。如果你喜欢这种设计模式,那么你会喜欢这个。此外,如果条目大于所需的零件尺寸,这也解决了空零件的问题。在最坏的情况下,您至少将拥有与条目一样多的部分。

在:

我的存档.zip

出去:

my-archive.part1of3.zip
my-archive.part2of3.zip
my-archive.part3of3.zip

注意:我正在使用日志记录和 Apache Commons FilenameUtils,但请随意使用工具包中的内容。

/**
 * Utility class to split a zip archive into parts (not volumes)
 * by attempting to fit as many entries into a single part before
 * creating a new part. If a part would otherwise be empty because
 * the next entry won't fit, it will be added anyway to avoid empty parts.
 *
 * @author Eric Draken, 2019
 */
public class Zip
{
    private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;

    private static final String ZIP_PART_FORMAT = "%s.part%dof%d.zip";

    private static final String EXT = "zip";

    private static final Logger logger = LoggerFactory.getLogger( MethodHandles.lookup().lookupClass() );

    /**
     * Split a large archive into smaller parts
     *
     * @param zipFile             Source zip file to split (must end with .zip)
     * @param outZipFile          Destination zip file base path. The "part" number will be added automatically
     * @param approxPartSizeBytes Approximate part size
     * @throws IOException Exceptions on file access
     */
    public static void splitZipArchive(
        @NotNull final File zipFile,
        @NotNull final File outZipFile,
        final long approxPartSizeBytes ) throws IOException
    {
        String basename = FilenameUtils.getBaseName( outZipFile.getName() );
        Path basePath = outZipFile.getParentFile() != null ? // Check if this file has a parent folder
            outZipFile.getParentFile().toPath() :
            Paths.get( "" );
        String extension = FilenameUtils.getExtension( zipFile.getName() );
        if ( !extension.equals( EXT ) )
        {
            throw new IllegalArgumentException( "The archive to split must end with ." + EXT );
        }

        // Get a list of entries in the archive
        try ( ZipFile zf = new ZipFile( zipFile ) )
        {
            // Silliness check
            long minRequiredSize = zipFile.length() / 100;
            if ( minRequiredSize > approxPartSizeBytes )
            {
                throw new IllegalArgumentException(
                    "Please select a minimum part size over " + minRequiredSize + " bytes, " +
                        "otherwise there will be over 100 parts."
                );
            }

            // Loop over all the entries in the large archive
            // to calculate the number of parts required
            Enumeration<? extends ZipEntry> enumeration = zf.entries();
            long partSize = 0;
            long totalParts = 1;
            while ( enumeration.hasMoreElements() )
            {
                long nextSize = enumeration.nextElement().getCompressedSize();
                if ( partSize + nextSize > approxPartSizeBytes )
                {
                    partSize = 0;
                    totalParts++;
                }
                partSize += nextSize;
            }

            // Silliness check: if there are more parts than there
            // are entries, then one entry will occupy one part by contract
            totalParts = Math.min( totalParts, zf.size() );

            logger.debug( "Split requires {} parts", totalParts );
            if ( totalParts == 1 )
            {
                // No splitting required. Copy file
                Path outFile = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, 1, 1 )
                );
                Files.copy( zipFile.toPath(), outFile );
                logger.debug( "Copied {} to {} (pass-though)", zipFile.toString(), outFile.toString() );
                return;
            }

            // Reset
            enumeration = zf.entries();

            // Split into parts
            int currPart = 1;
            ZipEntry overflowZipEntry = null;
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                Path outFilePart = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, currPart++, totalParts )
                );
                overflowZipEntry = writeEntriesToPart( overflowZipEntry, zf, outFilePart, enumeration, approxPartSizeBytes );
                logger.debug( "Wrote {}", outFilePart );
            }
        }
    }

    /**
     * Write an entry to the to the outFilePart
     *
     * @param overflowZipEntry    ZipEntry that didn't fit in the last part, or null
     * @param inZipFile           The large archive to split
     * @param outFilePart         The part of the archive currently being worked on
     * @param enumeration         Enumeration of ZipEntries
     * @param approxPartSizeBytes Approximate part size
     * @return Overflow ZipEntry, or null
     * @throws IOException File access exceptions
     */
    private static ZipEntry writeEntriesToPart(
        @Nullable ZipEntry overflowZipEntry,
        @NotNull final ZipFile inZipFile,
        @NotNull final Path outFilePart,
        @NotNull final Enumeration<? extends ZipEntry> enumeration,
        final long approxPartSizeBytes
    ) throws IOException
    {
        try (
            ZipOutputStream zos =
                new ZipOutputStream( new FileOutputStream( outFilePart.toFile(), false ) )
        )
        {
            long partSize = 0;
            byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                ZipEntry entry = overflowZipEntry != null ? overflowZipEntry : enumeration.nextElement();
                overflowZipEntry = null;

                long entrySize = entry.getCompressedSize();
                if ( partSize + entrySize > approxPartSizeBytes )
                {
                    if ( partSize != 0 )
                    {
                        return entry;    // Finished this part, but return the dangling ZipEntry
                    }
                    // Add the entry anyway if the part would otherwise be empty
                }
                partSize += entrySize;
                zos.putNextEntry( entry );

                // Get the input stream for this entry and copy the entry
                try ( InputStream is = inZipFile.getInputStream( entry ) )
                {
                    int bytesRead;
                    while ( (bytesRead = is.read( buffer )) != -1 )
                    {
                        zos.write( buffer, 0, bytesRead );
                    }
                }
            }
            return null;    // Finished splitting
        }
    }
于 2019-04-29T22:12:29.127 回答
0

下面的代码是我根据所需大小将目录结构中的 zip 文件拆分为块的解决方案。我发现以前的答案很有用,所以想用类似但更简洁的方法做出贡献。这段代码可以满足我的特定需求,我相信还有改进的余地。

import java.io.*;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;

class ChunkedZip {
    private final static long MAX_FILE_SIZE = 1000 * 1000 * 1024; //  around 1GB 
    private final static String zipCopyDest = "C:\\zip2split\\copy";

    public static void splitZip(String zipFileName, String zippedPath, String coreId) throws IOException {

        System.out.println("process whole zip file..");
        FileInputStream fis = new FileInputStream(zippedPath);
        ZipInputStream zipInputStream = new ZipInputStream(fis);
        ZipEntry entry = null;
        int currentChunkIndex = 0;
        //using just to get the uncompressed size of the zipentries
        long entrySize = 0;
        ZipFile zipFile = new ZipFile(zippedPath);
        Enumeration enumeration = zipFile.entries();

        String copDest = zipCopyDest + "\\" + coreId + "_" + currentChunkIndex + ".zip";

        FileOutputStream fos = new FileOutputStream(new File(copDest));
        BufferedOutputStream bos = new BufferedOutputStream(fos);
        ZipOutputStream zos = new ZipOutputStream(bos);
        long currentSize = 0;

        try {
            while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {

                ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
                System.out.println(zipEntry.getName());
                System.out.println(zipEntry.getSize());
                entrySize = zipEntry.getSize();

                ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                //long entrySize = entry.getCompressedSize();
                //entrySize = entry.getSize(); //gives -1

                if ((currentSize + entrySize) > MAX_FILE_SIZE) {
                    zos.close();
                    //construct a new stream
                    //zos = new ZipOutputStream(new FileOutputStream(new File(zippedPath, constructCurrentPartName(coreId))));
                    currentChunkIndex++;
                    zos = getOutputStream(currentChunkIndex, coreId);
                    currentSize = 0;

                } else {
                    currentSize += entrySize;
                    zos.putNextEntry(new ZipEntry(entry.getName()));
                    byte[] buffer = new byte[8192];
                    int length = 0;
                    while ((length = zipInputStream.read(buffer)) > 0) {
                        outputStream.write(buffer, 0, length);
                    }

                    byte[] unzippedFile = outputStream.toByteArray();
                    zos.write(unzippedFile);
                    unzippedFile = null;
                    outputStream.close();
                    zos.closeEntry();
                }
                //zos.close();
            }
        } finally {
            zos.close();
        }
    }

    public static ZipOutputStream getOutputStream(int i, String coreId) throws IOException {
        System.out.println("inside of getOutputStream()..");
        ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipCopyDest + "\\" + coreId + "_" + i + ".zip"));
        // out.setLevel(Deflater.DEFAULT_COMPRESSION);
        return out;
    }

    public static void main(String args[]) throws IOException {
        String zipFileName = "Large_files_for_testing.zip";
        String zippedPath = "C:\\zip2split\\Large_files_for_testing.zip";
        String coreId = "Large_files_for_testing";
        splitZip(zipFileName, zippedPath, coreId);
    }
}
于 2018-07-06T16:10:21.680 回答
0

这是我的解决方案:

public abstract class ZipHelper {

    public static NumberFormat formater = NumberFormat.getNumberInstance(new Locale("pt", "BR"));

    public static List<Path> zip(Collection<File> inputFiles, long maxSize) throws IOException {

        byte[] buffer = new byte[1024];
        int count = 0;
        long currentZipSize = maxSize;
        List<Path> response = new ArrayList<>();
        ZipOutputStream zip = null;
        for (File currentFile : inputFiles) {
            long nextFileSize = currentFile.length();
            long predictedZipSize = currentZipSize + nextFileSize;
            boolean needNewFile = predictedZipSize >= maxSize;
            System.out.println("[=] ZIP current (" + formater.format(currentZipSize) + ") + next file (" + formater.format(nextFileSize) + ") = predicted (" + formater.format(predictedZipSize) + ") > max (" + formater.format(maxSize) + ") ? " + needNewFile);
            if (needNewFile) {
                safeClose(zip);
                Path tmpFile = Files.createTempFile("teste-", (".part." + count++ + ".zip"));
                System.out.println("[#] Starting new file: " + tmpFile);
                zip = new ZipOutputStream(Files.newOutputStream(tmpFile));
                zip.setLevel(Deflater.BEST_COMPRESSION);
                response.add(tmpFile);
                currentZipSize = 0;
            }
            ZipEntry zipEntry = new ZipEntry(currentFile.getName());
            System.out.println("[<] Adding to ZIP: " + currentFile.getName());
            zip.putNextEntry(zipEntry);
            FileInputStream in = new FileInputStream(currentFile);
            zip.write(in.readAllBytes());
            zip.closeEntry();
            safeClose(in);
            long compressed = zipEntry.getCompressedSize();
            System.out.println("[=] Compressed current file: " + formater.format(compressed));
            currentZipSize += zipEntry.getCompressedSize();
        }
        safeClose(zip);
        return response;
    }

    public static void safeClose(Closeable... closeables) {
        if (closeables != null) {
            for (Closeable closeable : closeables) {
                if (closeable != null) {
                    try {
                        System.out.println("[X] Closing: (" + closeable.getClass() + ") - " + closeable);
                        closeable.close();
                    } catch (Throwable ex) {
                        System.err.println("[!] Error on close: " + closeable);
                        ex.printStackTrace();
                    }
                }
            }
        }
    }
}

和控制台输出:

[?] Files to process: [\data\teste\TestFile(1).pdf, \data\teste\TestFile(2).pdf, \data\teste\TestFile(3).pdf, \data\teste\TestFile(4).pdf, \data\teste\TestFile(5).pdf, \data\teste\TestFile(6).pdf, \data\teste\TestFile(7).pdf]
[=] ZIP current (3.145.728) + next file (1.014.332) = predicted (4.160.060) > max (3.145.728) ? true
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip
[<] Adding to ZIP: TestFile(1).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3d99d22e
[=] Compressed current file: 940.422
[=] ZIP current (940.422) + next file (1.511.862) = predicted (2.452.284) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(2).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@49fc609f
[=] Compressed current file: 1.475.178
[=] ZIP current (2.415.600) + next file (2.439.287) = predicted (4.854.887) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@cd2dae5
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip
[<] Adding to ZIP: TestFile(3).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@4973813a
[=] Compressed current file: 2.374.718
[=] ZIP current (2.374.718) + next file (2.385.447) = predicted (4.760.165) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@6321e813
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip
[<] Adding to ZIP: TestFile(4).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@79be0360
[=] Compressed current file: 2.202.203
[=] ZIP current (2.202.203) + next file (292.918) = predicted (2.495.121) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(5).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@22a67b4
[=] Compressed current file: 230.491
[=] ZIP current (2.432.694) + next file (4.197.512) = predicted (6.630.206) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@57855c9a
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip
[<] Adding to ZIP: TestFile(6).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3b084709
[=] Compressed current file: 3.020.115
[=] ZIP current (3.020.115) + next file (1.556.237) = predicted (4.576.352) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@3224f60b
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip
[<] Adding to ZIP: TestFile(7).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@63e2203c
[=] Compressed current file: 1.460.566
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@1efed156
[>] Generated ZIP files(s): [C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip, C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip, C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip, C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip, C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip]
于 2021-01-21T14:56:30.833 回答