Ytdyklly

I'm going to tell you exactly what I need in order to clarify the cryptic question in the title. I'm currently making scheduled MySQL backups of all my databases with something like:

mysqldump ... | gzip -c > mysql-backup.gz

This is ok, but I'm willing to make a separate file for each single database, since that will make it easier to take a look at dumped data or restore a single database:

for db in $dbs; do mysqldump ... $db | gzip -c > mysql-backup-$db.gz; done

I'd like to store all of the dumps for each single backup in a single .tar file, i.e. mysql-backup.tar.gz with all the dumped databases inside. I know I can simply leave .sql files uncompressed and then tar -cz *.sql, but 1) I'm searching for a way that doesn't need to temporarly store big files. In my current script, in fact, mysqldump is piped into gzip, so no big file is created.

2) Is there a similar way in which I can create .tar.gz from stdin?

3) Is tar -c *.sql.gz eqivalent to tar -cz *.sql?

Not easily. tar records not only file contents, but also file metadata (name, timestamps, permissions, owner and such). That information has to come from somewhere, and it won't be there in a pipe.

You could gzip your database dumps to a file (probably named for the database in question), append the file to a tar archive and then delete the file before proceeding to the next database. That'd result in a .gz.tar file, which is unusual but in no way a problem, and probably not use significantly more disk than gzipping a whole-database dump (it will be a little less efficiently compressed since it can't share across database borders).

I cobbled together some python to do what you want. It uses python's tarfile library to append stdin to a tar file, and then simply seeks back in the tar to rewrite the header with the right size at eof. The usage would be:

rm -f mytar

for db in $dbs

do mysqldump ... $db | gzip -c |

   tarappend -t mytar -f mysql-backup-$db.gz

done

tar tvf mytar

Here's the tarappend python script:

#!/usr/bin/python

# concat stdin to end of tar file, with given name. meuh on stackexchange

# $Id: tarappend,v 1.3 2015/07/08 11:31:18 meuh $



import sys, os, tarfile, time, copy

from optparse import OptionParser

try:

    import grp, pwd

except ImportError:

    grp = pwd = None



usage = """%prog: ... | %prog -t tarfile -f filename

Appends stdin to tarfile under the given arbitrary filename.

tarfile is created if it does not exist.

"""



def doargs():

    parser = OptionParser(usage=usage)

    parser.add_option("-f", "--filename", help="filename to use")

    parser.add_option("-t", "--tarfile", help="existing tar archive")

    (options, args) = parser.parse_args()

    if options.filename is None or options.tarfile is None:

        parser.error("need filename and tarfile")

    if len(args):

        parser.error("unknown args: "+" ".join(args))

    return options



def copygetlen(fsrc, fdst):

    """copy data from file-like object fsrc to file-like object fdst. return len"""

    totlen = 0

    while 1:

        buf = fsrc.read(16*1024)

        if not buf:

            return totlen

        fdst.write(buf)

        totlen += len(buf)



class TarFileStdin(tarfile.TarFile):

    def addstdin(self, tarinfo, fileobj):

        """Add stdin to archive. based on addfile() """

        self._check("aw")

        tarinfo = copy.copy(tarinfo)

        buf = tarinfo.tobuf(self.format, self.encoding, self.errors)

        bufoffset = self.offset

        self.fileobj.write(buf)

        self.offset += len(buf)



        tarinfo.size = copygetlen(fileobj, self.fileobj)

        blocks, remainder = divmod(tarinfo.size, tarfile.BLOCKSIZE)

        if remainder > 0:

            self.fileobj.write(tarfile.NUL * (tarfile.BLOCKSIZE - remainder))

            blocks += 1

        self.offset += blocks * tarfile.BLOCKSIZE

        # rewrite header with correct size

        buf = tarinfo.tobuf(self.format, self.encoding, self.errors)

        self.fileobj.seek(bufoffset)

        self.fileobj.write(buf)

        self.fileobj.seek(self.offset)

        self.members.append(tarinfo)



class TarInfoStdin(tarfile.TarInfo):

    def __init__(self, name):

        if len(name)>100:

            raise ValueError(name+": filename too long")

        if name.endswith("/"):

            raise ValueError(name+": is a directory name")

        tarfile.TarInfo.__init__(self, name)

        self.size = 99

        self.uid = os.getuid()

        self.gid = os.getgid()

        self.mtime = time.time()

        if pwd:

            self.uname = pwd.getpwuid(self.uid)[0]

            self.gname = grp.getgrgid(self.gid)[0]



def run(tarfilename, newfilename):

    tar = TarFileStdin.open(tarfilename, 'a')

    tarinfo = TarInfoStdin(newfilename)

    tar.addstdin(tarinfo, sys.stdin)

    tar.close()



if __name__ == '__main__':

    options = doargs()

    run(options.tarfile, options.filename)

No, and I miss that feature so much: my question on Ask Ubuntu.

If the file to be archived is a raw file with no filesystem metadata associated to it, tar doesn't have neither a filename nor a path necessary to build the internal directories / files tree (to say the least).

I think that something can be done in Perl, which has some library dedicated to compression / decompression / archiving of files: see if you can get the most out of this answer: a related answer on Ask Ubuntu.

You could consider using the tardy tar post-processor.

However, you might question the use of tar and consider some other ways to archive your things. In particular consider rsync and afio

Notice that mysqldump understands the --export-all option (see this). You might pipe that into some script understanding the boundaries, etc...