A few days ago, just after installing some new software, I ran into a curious bug that prevented one of my favorite programs from loading after Software X had been installed (names have been changed to protect the guilty).¬† The error I was getting way up in userland informed me that a driver could not load due to the fact that it failed to register one or more system notification routines (their words, not mine).¬† This was news to me, and in fact, I wasn't even sure what it was telling me initially, so I decided to do a little troubleshooting and see if it was something I could fix on my own.
If you do a Google search for "System notification routine," the first two results which come up, as documented by MS are PsSetLoadImageNotifyRoutine and PsSetCreateProcessNotifyRoutine.¬† Per the documentation here and here:
The PsSetLoadImageNotifyRoutine routine registers a driver-supplied callback that is subsequently notified whenever an image is loaded (or mapped into memory).
The PsSetCreateProcessNotifyRoutine routine adds a driver-supplied callback routine to, or removes it from, a list of routines to be called whenever a process is created or deleted.
So at a glance, we've at least found the two calls that are likely failing (for unknown reasons), and it's most likely that this is a compatibility issue that cannot be resolved by the user, but it never hurts to try!¬† In reading the documentation about PsSetCreateProcessNotifyRoutine, the folks at Microsoft also note that "the system can register up to 64 process-creation callbacks."¬† Based on the error message, my initial hypothesis was that I had reached the system limit which would explain why uninstalling other pieces of software made the problem go away.¬† After discussing this with some other folks, it seemed unlikely, but it was really all I had to go with which means the next task was to enumerate all registered process-creation callbacks to either confirm or eliminate this as the cause.
When all you have is a hammer, everything looks like a nail.
My initial approach was to
avoid heading straight for the debugger find a lazy solution using either process monitor or Volatility, neither of which ended up doing what I needed.¬† Since I think it's important to document failures as well as successes, let's quickly cover why Volatility didn't do what I wanted.¬† First, it was my fault for using a tool for something other than its intended purpose and expecting good results (I should have known better after this long in the security industry...).¬† Second, I used the "enumfunc" plugin from Volatility 2.2 which did exactly what it was supposed to do, but ended up not being what I needed. The closest this output got me was:
<KERNEL> Import aswVmm.sys 904 0x000000008062deb2 ntoskrnl.exe!PsSetCreateProcessNotifyRoutine
While that's very helpful, the numbers didn't add up when it came to imported functions so I couldn't see anything definitive based on the output.¬† Lastly, the memory image I was supplying Volatility with was a 64-bit version of Windows 7 SP1 which isn't supported by all of the plugins such as "callbacks" and others which may have helped me put this to bed a long time ago.
In the end, kd.exe was the only way out of this and thankfully, I found a Windbg/KD script written by Scott Noone back in November of 2010 that did exactly what I wanted.¬† After configuring the machines for kernel debugging over a 1394 cable, I ran the script and sure enough, it worked flawlessly.¬† Unfortunately, it turns out only 15 process creation-callbacks were registered on the system, so I quickly went from having a hypothesis, to having no idea =/¬† However, on the bright side, I did get to learn a bit and I found the answer to my question, which was how to enumerate process-creation callbacks!¬† So, if anyone comes here looking for ways to enumerate callbacks or list registered notification routines, this post should get you started.
In the mean time, I'll see if I can identify why this driver still fails to load and post any additional findings.