Another issue where scripting easily helps your day job is handling the copious amounts of log files. I handle about 15 web servers in our production environment (that I remember off the top of my head), and quite a few in our dev/stage/qa environments. These generate a lot of log files. Our production environment is the one that is most critical. If the logs fill the drive, the site stops working, something we don’t like happening, and has happened on two separate occasions.
One of our IIS servers generates a 300MB log file a day for the activity on the web site. After a few days, this gets pretty large, pretty quick. So I had to figure a way of handling the clean up of these log files. As I wanted to analyze them, I need to keep them, so zip was the best option. Unfortunately, again, this would be a one liner in linux, but DOS/Windows makes this a little more complicated. I could knock together a small application in Delphi/VB to do the work for me, but that’d waist time, and require me to maintain yet another application. So I opted for a DOS script, and WinRAR.
My DOS batch scripting is a little rusty, so there may be an easier way to handle this, but I ended up with two scripts. The first script performs the loop, while the second script performs the compression.
@echo off
for %%a in ("*.log") do call _inner.bat %%a
This is fairly clear. For every log file in the active directory, it calls a file called _inner.bat with the argument of the filename. Then there is the code for _inner.bat:
@echo off
set path="D:\Program Files\WinRar\";%path%
set _file=%1
set _newfile=%_file:log=zip%
rar m %_newfile% %_file%
This does two things here. It updates the %path% variable to include the path to WinRAR so I can use the files. It then grabs the name into a variable and renames from somename.log to somename.zip. The last step is calling rar with the m argument. This compresses, and removes the file once completed.
This is now setup as a scheduled job that runs every Sunday at 2am, and keeps our log folders down. Compression of a 300MB text file gets it down into the 1-2MB range. Next step is a job to pull them down to the corporate office for web log analysis.