Kevin J. Cummings wrote: > On 05/26/2010 01:16 PM, Rector, David wrote: >> Hello, >> >> I have studied various filesystems, and am fairly familiar with how they are structured. However, I am currently stuck on trying to do what seems like a simple thing. >> >> I would like to join two files together without having to physically copy bytes (i.e. I have vary large files, so I don't want to use 'cat'). It seems to me that it should be possible to simply modify the file entry in the filesystem such that the last inode of the first file points to the first inode of the second file. I guess this is similar to a "hard link", but used to join files rather than simply have another pointer to one file. >> >> I have seen 'mmv' and 'lxsplit' and they all seem to do the same thing, namely they want to physically copy the bytes in order to join two files together. >> >> Is there any such utility in linux to perform such a hard link to join or connect two files together without having to copy bytes? > > If you could guarantee that the last extent used by the first file was > completely full of data with no extraneous bytes, it might be possible > to "merge" the extent maps of the 2 files into a single file entry. If > you cannot guarantee that, then you will have to copy bytes from the 2nd > file to the end of the first file. But everything becomes possible if the fileystem permits partially empty blocks in the middle of the file. No filesystem does it AFAIK, but it is not a big issue, as partial blocks (or compacted tails) are already permitted at the end of the file. New filesystems use extents rather than blocks, so if the extents are measured in bytes instead of 512b-blocks you can just use a smaller extent in the middle of the file where the join happened. At this point, you can support inplace-joining, inplace-inflating (add 10000 bytes in this file at position 300000), inplace-erasure (remove 10000 bytes at position 300000) and data shuffling (swap the first 50meg of the file with the last 50meg). With heavy usage you have just created a new kind of fragmentation, which can be corrected with the usual defragmentation tools (including "cp"). (add that fragmentation is losing importance with the spreading of SSD) Considering that sparse files have been a reality for decades and that the implementation of operation with inside-file byte-grained extents is not more difficult than truncate, I wonder if we will see something of this kind in some advanced filesystem (btrfs?). There are a lot of possible uses: - delete/replace mail in mbox format repositories - smart packaging (delete from tar, delete from zip) - in-place iso creation and.... just imagine..... - video editing (!) add/remove/replace frames inside a 150GiB captured video Where can you submit ideas to btrfs? It also has COW, so everything becomes even more exciting... -- Roberto Ragusa mail at robertoragusa.it -- users mailing list users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe or change subscription options: https://admin.fedoraproject.org/mailman/listinfo/users Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
