Wednesday, April 6, 2011

Event ID 2019 - NonPaged Pool exhaustion…

Having these problems with one of our development servers – Windows Server 2003 Enterprise SP2 (x32), Dual Xeon X5550, 24 GB RAM (PAE), hosting both users (Citrix 4.5.7) and a number of database servers – for a few months now, with occasional hard stops due to Event ID 2019 every few days, we finally got a lead…

We tried scheduling poolmon to run every minute (poolmon.exe -p -b -s -r -e -n %OUTDIR%\poolmon.%ISODATE%.log) without much luck - we’ve seen the depletion of nonpaged pool (for this Windows version = 256 MB) from one minute to the other, and with resources gone poolmon would not be able to run at all, producing an empty log file with just the header information.

Finally the other day noticing the usual slowdown then looking in Task Manager > Performance (tab) > Kernel Memory (area) > NonPaged value was very high at 259.xxx KB, quickly ran the poolmon which failed to run at first attempt, but on the second run we finally got the offender, pooltag Ica.

Not sure yet whether this relates to Citrix ICA or Microsoft Terminal Services, but personally I feel so relieved - after applying all sorts of hotfixes and blaming everything from the antivirus, the backup software, the fact that we had so many databases and users on the same machine (I know, not ideal but that was the only choice at the time)...

In normal conditions (peak times) the total NonPaged would be fairly stable at around 90-95 MB (out of 256 MB) with the top pooltag MFE0 (McAfee - Access Protection Filter Driver ?) at around 9 MB, and ICA (Microsoft - Terminal Server Driver ?) nowhere at the top with a mere ~150 KB. Then ICA pooltag takes a huge jump to 90 M when problems arise, second place MFE0 with a small increase to 13 MB.