Re: Use tar to append?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Mike McCarty wrote:
> Mikkel L. Ellertson wrote:
> 
>> The problem is that the entire archive is compressed, and not the
> 
> [snip]
> 
> No, it is not. Please speak of what you know, rather than
> what you conjecture.[1]
> 
Are you saying that this is not how compressed tar archives are
created? If so, where did you get this? When you use the z option,
tar filters through gzip. If you use the j option, it filters throug
bzip2. This is why you can convert a compressed tar archive to an
uncompressed archive with gzip or bzip2. For a large archive, this
take a fair amount of time. When you want to add to the end of a
compressed archive, it has uncompress the entire archive, add the
files, and compress the new archive. This is also why it is so hard
to recover files from a damaged compressed archive that are after
the damaged section, or from a damaged tape built this way.

Now, if you were restoring a file from the archive, instead of
adding to the archive, then you can stop uncompressing after you get
to the end of the file you are after.

> I ran a test in which I created a tar archive, not compressed,
> but straight tar format. The size of the file was 3653601280 bytes.
> 
> I created another tar archive, also not compressed straight
> tar format. The size of that file was 10240 bytes. It took
> less than a second to create.
> 
> I then used "tar A" to append the tiny archive to the larger
> one, and the run time was 2:20; that is two minutes twenty
> seconds. During that time, my machine was approx. 80% wait state,
> and approx. 16% system state, per top.

Try the same thing with the archives compressed, and let me know how
long it takes.
> 
> It may be that, due to the format of a tar file, tar is
> architecturally constrained to do individual seeks for
> every file contained in the archive, and that doing
> thousands of seeks in this rather large file is necessarily
> time consuming. This archive has 34619 files in it. I watched
> the file system size using df, and found that it did not vary
> during this process, so I conjecture that this was seeks
> rather than copies.
> 
Well, tar was created for use with tape drives, with limited to no
seek capabilities. It may well be seeking from the start of the
archive for each file. I do know that when restoring one file from a
tape, it will start at the beginning of the tape archive, and search
through until it gets to the file it wants. This can be interesting
with multi-tape archives.

>From the results you are seeing, it would appear that you are using
the wrong tool for the job you are trying to do.

Mikkel
-- 

  Do not meddle in the affairs of dragons,
for thou art crunchy and taste good with Ketchup!


[Index of Archives]     [Current Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [Yosemite Photos]     [KDE Users]     [Fedora Tools]     [Fedora Docs]

  Powered by Linux