Is it possible to automatically zip newly created files in a directory

Context

I am required to use a poorly designed java application that logs A LOT of information while it is running. Under standard usage, it will create 100s of MB of logs per hour.

I don't need historical logs and it currently seems that the logrotate utility can't keep up with it as it doesn't run frequently enough. The application is closed source and rotates it's own logs at around 36MB.

My Linux distribution is RHEL7.

Question

I'd like to reduce wasted space by compressing and rotating the logs.

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Is it possible to automatically delete all files in the format of assessor-cli.X.log where X is a digit greater than... say 5 (i.e. keep only the 5 most recent logs).

Here is my attempt at a logrotate file:

# cat /etc/logrotate.d/cis_assessor

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

        size 30M

This logrotate job would need to catch the log between the size of 30MB and 36MB to actually come into effect which might only be a 10 second period. That's why I'm asking about the manual path of compressing and deleting files without logrotate.

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

asked Feb 12 at 1:21

Crypteya

37717

1

You've mentioned that logrotate doesn't run frequently enough for your use case. It's possible to set up a logrotate command with a custom state file and custom configuration file so that runs independently of the system's logrotate. At that point, it's just a normal command you can schedule with cron.

– Haxiel
Feb 12 at 5:36

I wouldn't really want to set the cron job to run ever 5 seconds. Which is why I was looking for a tool that could monitor events in the directory (as the tool doesn't run all the time). This comment could definitely be of use to others though

– Crypteya
Feb 13 at 0:41

add a comment |

Context

I am required to use a poorly designed java application that logs A LOT of information while it is running. Under standard usage, it will create 100s of MB of logs per hour.

My Linux distribution is RHEL7.

Question

I'd like to reduce wasted space by compressing and rotating the logs.

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Is it possible to automatically delete all files in the format of assessor-cli.X.log where X is a digit greater than... say 5 (i.e. keep only the 5 most recent logs).

Here is my attempt at a logrotate file:

# cat /etc/logrotate.d/cis_assessor

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

        size 30M

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

asked Feb 12 at 1:21

Crypteya

37717

1

You've mentioned that logrotate doesn't run frequently enough for your use case. It's possible to set up a logrotate command with a custom state file and custom configuration file so that runs independently of the system's logrotate. At that point, it's just a normal command you can schedule with cron.

– Haxiel
Feb 12 at 5:36

I wouldn't really want to set the cron job to run ever 5 seconds. Which is why I was looking for a tool that could monitor events in the directory (as the tool doesn't run all the time). This comment could definitely be of use to others though

– Crypteya
Feb 13 at 0:41

add a comment |

Context

I am required to use a poorly designed java application that logs A LOT of information while it is running. Under standard usage, it will create 100s of MB of logs per hour.

My Linux distribution is RHEL7.

Question

I'd like to reduce wasted space by compressing and rotating the logs.

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Is it possible to automatically delete all files in the format of assessor-cli.X.log where X is a digit greater than... say 5 (i.e. keep only the 5 most recent logs).

Here is my attempt at a logrotate file:

# cat /etc/logrotate.d/cis_assessor

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

        size 30M

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

asked Feb 12 at 1:21

Crypteya

37717

Context

I am required to use a poorly designed java application that logs A LOT of information while it is running. Under standard usage, it will create 100s of MB of logs per hour.

My Linux distribution is RHEL7.

Question

I'd like to reduce wasted space by compressing and rotating the logs.

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Is it possible to automatically delete all files in the format of assessor-cli.X.log where X is a digit greater than... say 5 (i.e. keep only the 5 most recent logs).

Here is my attempt at a logrotate file:

# cat /etc/logrotate.d/cis_assessor

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

        size 30M

linux gzip logrotate

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

asked Feb 12 at 1:21

Crypteya

37717

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

asked Feb 12 at 1:21

Crypteya

37717

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

edited Feb 17 at 1:40

Jeff Schaller

42.5k1158135

asked Feb 12 at 1:21

Crypteya

37717

asked Feb 12 at 1:21

Crypteya

37717

asked Feb 12 at 1:21

Crypteya

37717

1

You've mentioned that logrotate doesn't run frequently enough for your use case. It's possible to set up a logrotate command with a custom state file and custom configuration file so that runs independently of the system's logrotate. At that point, it's just a normal command you can schedule with cron.

– Haxiel
Feb 12 at 5:36

I wouldn't really want to set the cron job to run ever 5 seconds. Which is why I was looking for a tool that could monitor events in the directory (as the tool doesn't run all the time). This comment could definitely be of use to others though

– Crypteya
Feb 13 at 0:41

add a comment |

1

You've mentioned that logrotate doesn't run frequently enough for your use case. It's possible to set up a logrotate command with a custom state file and custom configuration file so that runs independently of the system's logrotate. At that point, it's just a normal command you can schedule with cron.

– Haxiel
Feb 12 at 5:36

I wouldn't really want to set the cron job to run ever 5 seconds. Which is why I was looking for a tool that could monitor events in the directory (as the tool doesn't run all the time). This comment could definitely be of use to others though

– Crypteya
Feb 13 at 0:41

You've mentioned that logrotate doesn't run frequently enough for your use case. It's possible to set up a logrotate command with a custom state file and custom configuration file so that runs independently of the system's logrotate. At that point, it's just a normal command you can schedule with cron.

– Haxiel
Feb 12 at 5:36

I wouldn't really want to set the cron job to run ever 5 seconds. Which is why I was looking for a tool that could monitor events in the directory (as the tool doesn't run all the time). This comment could definitely be of use to others though

– Crypteya
Feb 13 at 0:41

add a comment |

1 Answer
1

active

oldest

votes

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Yes, it is. Just target the newly created file with something that can watch for new files in a directory (like entr)

So you'll create a logrotate config like this (/etc/logrotate.d/newlogrotateconf)

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

}

Then you'll run entr in a loop on the directory to tie logrotate into inotify/epoll,

echo -n /usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/

  | ./entr -dnc logrotate --force /etc/logrotate.d/newlogrotateconf

edited Feb 12 at 2:55

Crypteya

37717

answered Feb 12 at 1:55

Evan Carroll

5,734114482

I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it.

– Crypteya
Feb 12 at 2:50

@Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files).

– Evan Carroll
Feb 12 at 2:57

1

Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :)

– Crypteya
Feb 12 at 3:03

add a comment |

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f500076%2fis-it-possible-to-automatically-zip-newly-created-files-in-a-directory%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Yes, it is. Just target the newly created file with something that can watch for new files in a directory (like entr)

So you'll create a logrotate config like this (/etc/logrotate.d/newlogrotateconf)

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

}

Then you'll run entr in a loop on the directory to tie logrotate into inotify/epoll,

echo -n /usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/

  | ./entr -dnc logrotate --force /etc/logrotate.d/newlogrotateconf

edited Feb 12 at 2:55

Crypteya

37717

answered Feb 12 at 1:55

Evan Carroll

5,734114482

I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it.

– Crypteya
Feb 12 at 2:50

@Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files).

– Evan Carroll
Feb 12 at 2:57

1

Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :)

– Crypteya
Feb 12 at 3:03

add a comment |

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Yes, it is. Just target the newly created file with something that can watch for new files in a directory (like entr)

So you'll create a logrotate config like this (/etc/logrotate.d/newlogrotateconf)

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

}

Then you'll run entr in a loop on the directory to tie logrotate into inotify/epoll,

echo -n /usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/

  | ./entr -dnc logrotate --force /etc/logrotate.d/newlogrotateconf

edited Feb 12 at 2:55

Crypteya

37717

answered Feb 12 at 1:55

Evan Carroll

5,734114482

I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it.

– Crypteya
Feb 12 at 2:50

@Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files).

– Evan Carroll
Feb 12 at 2:57

1

Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :)

– Crypteya
Feb 12 at 3:03

add a comment |

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Yes, it is. Just target the newly created file with something that can watch for new files in a directory (like entr)

So you'll create a logrotate config like this (/etc/logrotate.d/newlogrotateconf)

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

}

Then you'll run entr in a loop on the directory to tie logrotate into inotify/epoll,

echo -n /usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/

  | ./entr -dnc logrotate --force /etc/logrotate.d/newlogrotateconf

edited Feb 12 at 2:55

Crypteya

37717

answered Feb 12 at 1:55

Evan Carroll

5,734114482

As the app already splits out the logs into new files, is it possible to automatically compress newly created files in a directory?

Yes, it is. Just target the newly created file with something that can watch for new files in a directory (like entr)

So you'll create a logrotate config like this (/etc/logrotate.d/newlogrotateconf)

/usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/assessor-cli.log {

        missingok

        notifempty

        compress

        rotate 5

}

Then you'll run entr in a loop on the directory to tie logrotate into inotify/epoll,

echo -n /usr/share/foreman-proxy/Ansible/CIS/audit/Assessor-CLI-4.0.2/logs/

  | ./entr -dnc logrotate --force /etc/logrotate.d/newlogrotateconf

edited Feb 12 at 2:55

Crypteya

37717

answered Feb 12 at 1:55

Evan Carroll

5,734114482

edited Feb 12 at 2:55

Crypteya

37717

edited Feb 12 at 2:55

Crypteya

37717

edited Feb 12 at 2:55

Crypteya

37717

answered Feb 12 at 1:55

Evan Carroll

5,734114482

answered Feb 12 at 1:55

Evan Carroll

5,734114482

answered Feb 12 at 1:55

Evan Carroll

5,734114482

I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it.

– Crypteya
Feb 12 at 2:50

@Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files).

– Evan Carroll
Feb 12 at 2:57

1

Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :)

– Crypteya
Feb 12 at 3:03

add a comment |

I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it.

– Crypteya
Feb 12 at 2:50

@Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files).

– Evan Carroll
Feb 12 at 2:57

1

Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :)

– Crypteya
Feb 12 at 3:03

I'm not able to install that package in my environment as in rhel/centos it's under the epel repository and my organisation doesn't allow it. But I see what you've done and it looks good. For now I've had a chat with my team and will just disable logging (or set it to only record errors) for the offending app. I'm accepting this answer as it could help someone who finds it.

– Crypteya
Feb 12 at 2:50

@Crypteya see about inotifywait if that's available. If not, you may consider compiling entr (it's like 3 files).

– Evan Carroll
Feb 12 at 2:57

Thanks, I checked that package as well after reading the other answers in your linked question. Same deal. I'll almost certainly compile entr if the reduced logging becomes unsuitable. Thanks again :)

– Crypteya
Feb 12 at 3:03

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Unix & Linux Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Ytdyklly