Speed up collection scanning

Hi, malware collectors of the world!

Nowadays one of the problems that collectors have is the required amount of time to generate new logs. Today I will discuss several methods to speed up collection scanning times.

In the past virus collections used to take around 200 or 300 MB. With that size it was possible to generate new logs every day, even using several antivirus.

After year 2000 the amount of samples started to increase heavily and collectors began to generate new logs weekly instead of daily. At the same time all the antivirus used to exchange were dropped and only KAV remained, being the standard antivirus to exchange.

Actually KAV is still the standard antivirus for malware exchange as I commented in other post. So apart of generic ways, I will focus in methods to speed up KAV scanning.

1.- The most obvious way to boost things is to use the best available hardware. The Intel Core i7 is a good choice. The amount of RAM is not so important but a fast H.D. is.

2.- An even more obvious way to speed up log creation is to use several computers. Just share the task load between several computers.

3.- If you are creating logs to trade scan only your exchange collection.

The exchange collection will be formed by unique samples. Don´t keep several copies of the same identified sample.

4.- Something that slows down KAV very much are the packed samples, so unpack all possible packed samples.

Extract detected files from setups/installations, embedded and dropper files.

Examples of that kind of samples are setups created with: NSIS, Setup Factory, autoextractable files (RAR, ZIP, ...), etc.

Don´t extract compressed files. I mean files packed with UPX, Armadillo, Themida, MEW, etc. Only extract that kind of files when a setup or installation file is compressed with any of them.

You will recognize what stuff you must unpack looking at KAV log. Here you can see some examples of the kind of stuff you should process:

c:\test\ASTRUM.EX_/data0004 Infected Backdoor.IRC.Seiseni 
c:\test\ASTRUM.EX_/data0008 Infected Backdoor.IRC.Seiseni 
c:\test\ASTRUM.EX_/data0009 Infected not-a-virus:Client-IRC.Win32.mIRC.601 
c:\test\HMIMYS.EX_/123.exe Infected Backdoor.Win32.Hupigon.ejub 
c:\test\INIT1.EX_/data0000 Infected Trojan.Win32.Chinaad.ni 
c:\test\INIT2.EX_/data0000 Infected Trojan.Win32.Chinaad.ne 
c:\test\INNO.EX_/file19 Infected not-a-virus:FraudTool.Win32.AntiSpywareSoldier.b 
c:\test\INNO2.EX_/data0032 Infected not-a-virus:Monitor.Win32.ParentsFriend.a 
c:\test\INSTYLER.EX_/astem.as Infected Backdoor.IRC.Zapchast 
c:\test\INSTYLER.EX_/bstem.as Infected Backdoor.IRC.Zcrew 
c:\test\INSTYLER.EX_/oystem.er Infected Backdoor.IRC.Zcrew 
c:\test\KAOS.EX_/data0000.cab/2.exe Infected Backdoor.Win32.Hupigon.ehnx 
c:\test\MSC.EX_/MSC.EX_ Infected Trojan-Downloader.Win32.Banload.ddh 
c:\test\NBINDER1.EX_/ppp.exe Infected Backdoor.Win32.Turkojan.bkn 
c:\test\NBINDER2.EX_/testxxx4.exe/rbot2.exe Infected Backdoor.Win32.Rbot.wnl 
c:\test\NBINDER3.EX_/server.exe-crypted.exe Infected Trojan-Dropper.Win32.VB.azv 
c:\test\NBINDER4.EX_/svchost.exe Infected Backdoor.Win32.SdBot.ewp 
c:\test\NBINDER5.EX_/crypted1.exe Infected Backdoor.Win32.Bifrose.uzu 
c:\test\NBINDER6.EX_/dl.exe Infected Trojan-Downloader.Win32.Agent.ahbi 
c:\test\NSIS.EX_/data0002 Infected Backdoor.Win32.Visel.afy 
c:\test\NSPACKER.EX_/data0000.cab/SERVER~1.EXE Infected Backdoor.Win32.Hupigon.dsx 
c:\test\ORIEN.EX_/data0000.cab/SERVER~1.EXE Infected Backdoor.Win32.Hupigon.dsx 
c:\test\ORIEN2.EX_/data0000.cab/7.exe Infected Trojan-GameThief.Win32.OnLineGames.tkws 
c:\test\PCGUARD1.EX_/data0000.cab/server.exe Infected Trojan.Win32.Midgare.aamx 
c:\test\PCGUARD2.EX_/data0000.cab/server.exe Infected Trojan.Win32.Midgare.aadg 
c:\test\QBFC.EX_/1 Infected Flooder.Win32.Assault.10 
c:\test\QBFC2.EX_/0 Infected Backdoor.Win32.Netbus.170 
c:\test\RAP.EX_/rinst.exe Infected Trojan.Win32.KillAV.dt 
c:\test\SEA.EX_/setup.zip/1/ver.2/AUR.exe Infected IM-Flooder.Win32.AUR.c 
c:\test\SEA.EX_/setup.zip/5/HM_comC.exe Infected Trojan.Win32.Delf.kl 
c:\test\SEA.EX_/setup.zip/6/icq-brute.exe Infected HackTool.Win32.BruteForce.u 
c:\test\SEA.EX_/setup.zip/8/1.5.191_Pro/IPDbrute_1.5.191.exe Infected not-a-virus:PSWTool.Win32.IpdBrute.15 
c:\test\SEA.EX_/setup.zip/8/IPDbrute_2.0_Lite/IPDbrute_2.0_Lite.exe Infected not-a-virus:PSWTool.Win32.IpdBrute.20 
c:\test\SEA.EX_/setup.zip/8/IPDbrute_2.0_Pro_old/IPDbrute2.exe Infected not-a-virus:PSWTool.Win32.IpdBrute.20 
c:\test\SEA.EX_/setup.zip/11/recover.exe Infected not-a-virus:PSWTool.Win32.ICQ.y 
c:\test\SEA.EX_/setup.zip/12/UIC.exe Infected Flooder.Win32.Agent.bb 
c:\test\SEA.EX_/setup.zip/16 Infected not-a-virus:PSWTool.Win32.ICQ.v 
c:\test\SEA2.EX_/setup.zip/25 Infected not-a-virus:Client-IRC.Win32.mIRC.603 
c:\test\SEA2.EX_/setup.zip/26 Infected not-a-virus:RiskTool.Win32.HideWindows 
c:\test\SIM.EX_/data1 Infected Trojan.BAT.KillFiles.ge 
c:\test\SVKP.EX_/data0000.cab/4_BK_BK.exe Infected Packed.Win32.PolyCrypt.b 
c:\test\THINSTAL.EX_/AQ.exe Infected Trojan-Downloader.Win32.Small.akjq 
c:\test\UPACK.EX_/data0000.cab/lin2.exe Infected Trojan-Downloader.Win32.BHO.un 
c:\test\UPACK.EX_/data0000.cab/rmt-live.exe Infected Trojan.Win32.Inject.ihr 

5.- Don´t scan very old files.

Move apart old files (files you got from year 2002 and older) and don´t scan them every week. Maybe once per month will be enough.

Why this? Because KAV probably will not change the identification names of that samples, so the ID will remain equal week after week.

If anyone have any other trick to speed up log creation he will be welcome.