Ytdyklly

On my 3 TB external hard drive, I have a 2.7 TB directory containing relatively small files. I would like to compress this 2.7 TB directory and remove it to keep only the compressed version. The issue is that I do not have enough storage to first zip and then rm the non-zipped directory.

Is there a way around this problem or do I necessarily have to acquire more storage for the manipulation?

You can try the --remove-files argument to tar. Say you want to compress everything on directory FOO, you would:
tar -czf FooCompressed.tar.gz --remove-files FOO

Arguments explained:

• c: create TAR


• z: compress using GZIP, you can switch to -j for BZIP2 or -J for LZMA(xz)


• f: output to file instead of of STDOUT


• remove-files: self explanatory

If the largest file in the directory is less than 300GB (the amount of free space), the easiest option is to compress files individually rather than creating an archive; something like

find directory -type f ! -name '*.xz' -print 0 | xargs -0 xz -9

will compress all non-compressed files in directory using xz at compression level 9. This will replaces files one at a time with the corresponding compressed version, adding a .xz extension (so foo.txt becomes foo.txt.xz).

In fact this can work even if the largest file is larger than the available free space, as long as you have enough files smaller than the available free space: start by compressing the smaller files, which will progressively free more and more space up, until (hopefully) you have enough free space to handle the largest file.

I normally never make zip files, so I'm no expert, but it looks like zip by default just adds files to an archive, so something like (untested - contains an rm command = potentially dangerous - Test yourself before risking your data):

for f in $(ls --sort=size --reverse); do zip -9 archive.zip $f; rm $f; done

in the directory, could probably do it.

• c: create TAR


• z: compress using GZIP, you can switch to -j for BZIP2 or -J for LZMA(xz)


• f: output to file instead of of STDOUT


• remove-files: self explanatory

something happens on stage 1-3 and you lose all your original files on step 4