SQL Server 2012 Install Fails on Server Core 2008 R2 VHD - Object reference not set to an instance of an object.

So, I'm going through the Administering Microsoft SQL Server 2012 Databases book for the 70-462 certification exam. The only way, I could quickly get access to Windows Server Core for the test exercises was to download the VHD provided by Microsoft.

Well, strange thing. It is missing a critical registry key. Without this key, SQL Server fails with a horribly vague error. To make it more fun, while the description tells me to look in a file named summary.txt, there is no file named summary.txt anywhere.

The following error occurred:
Object reference not set to an instance of an object.

Error result: -2147467261
Result facility code: 0
Result error code: 16387

Please review the summary.txt log for further details

Slightly more - but also unhelpful - information can be found in the Component Updater log file generated by the installer.

Exception summary:
The following is an exception stack listing the exceptions in outermost to innermost order
Inner exceptions are being indented

Exception type: System.NullReferenceException
    Message: 
        Object reference not set to an instance of an object.
    Data: 
      DisableWatson = true
    Stack: 
        at Microsoft.SqlServer.Configuration.MsiExtension.ArpRegKey.CleanupPatchedProductRegistryInfo()
        at Microsoft.SqlServer.Configuration.MsiExtension.SetPatchInstallStateAction.ExecuteAction(String actionId)
        at Microsoft.SqlServer.Chainer.Infrastructure.Action.Execute(String actionId, TextWriter errorStream)
        at Microsoft.SqlServer.Setup.Chainer.Workflow.ActionInvocation.ExecuteActionHelper(TextWriter statusStream, ISequencedAction actionToRun, ServiceContainer context)

Anyway, after a couple of hours of messing around with it, I find a post on the Microsoft Forums that finally sheds some light on it.  Apparently the HKLM\Software\Microsoft\Windows NT\CurrentVersion\Uninstall registry key doesn't exist in the distributed virtual hard drive. Either adding this key or running an installer that creates it seems to fix the problem.

Hopefully, this will save others some debugging time and effort.

Configuring DEP on Windows Server 2008 R2 from a 32bit NSIS Installer - Revisited

Thanks to another blog, The Old New Thing, I found out there is another way to handle writing to the 64bit Registry from a 32bit program - without the needing to deal with file redirection.  Well, almost without the need for it. Starting with Windows Vista, a special alias %windir%\Sysnative was added to allow access to the System32 directory even when running a 32bit program.

Unfortunately, Windows XP and Windows Server 2003 don't have this feature; and since we still work with both of these versions of Windows, I cannot incorporate it into our code.  Anyway, since the original post is one of my more popular, I thought I would update the code.

var /GLOBAL NSISRegPath
StrCpy $NSISRegPath "SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

var /GLOBAL EXERegPath
StrCpy $EXERegPath "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

${IF} ${AtLeastWinVista}
    WriteRegStr HKLM "$NSISRegPath" "$INSTDIR\program.exe" "DisableNXShowUI"
    ${IF} ${RunningX64}
        ExecWait '$WINDIR\SysNative\reg.exe add "$EXERegPath" /v \
            "$INSTDIR\program.exe" /d "DisableNXShowUI"'
    ${ENDIF}
${ENDIF}

Note: Lines that end with a backslash represent long lines that have been wrapped.

Configuring DEP on Windows Server 2008 R2 from a 32bit NSIS Installer

If the title is hard to understand, let me just shorten it to this.  The woes of compatibility testing!

Included with the Platypus Billing System is a number of 3rd party ActiveX libraries.  Most of the time, these libraries are wondrous things.  Unfortunately, one, in particular, has a problem with Windows DEP.

Now, upon installation of the Platypus client, we can get around this by configuring the Application Compatibility settings of our executable to bypass DEP.  So, whenever the exe is launched, the OS will not trap DEP problems for our process.  We do all of this by simply writing to the appropriate location in the registry during the installation process.

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers

All the way from Windows XP up to Windows Server 2008, we simply had to write to the registry.  It didn't even matter whether the OS is x86 or x64.  It has just worked.

Anyway, one of the more ingenious and infamous features from Windows Vista is redirection.  You see, whenever a 32bit application writes to HKEY_LOCAL_MACHINE in the registry on 64bit Windows, it is actually writing to HKEY_LOCAL_MACHINE\Software\Wow6432Node. So, whenever our software was installed, because the installation set is a 32bit application itself, when it writes to the registry, it is actually writing to the 32bit subset of the registry.  This includes those pesky Application Compatibility settings I mentioned.  Up until the latest version of Windows Server, the OS didn't care whether you configure the compatibility settings in the 32bit or 64bit registry.  It checked both when the process started.  Here's an example of code we used for configuring DEP*.

!include LogicLib.nsh
!include WinVer.nsh

var /GLOBAL NSISRegPath
StrCpy $NSISRegPath "SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

${IF} ${AtLeastWinXP}
    WriteRegStr HKLM "$NSISRegPath" "$INSTDIR\program.exe" "DisableNXShowUI"
${ENDIF}

Starting with Windows Server 2008 R2, Application Compatibility settings in the 32bit registry now appear to be ignored.  Meaning the Platypus client will now crash whenever our troublesome ActiveX library rears its head, because our installation set can't get to the 64bit registry.  Not easily, anyway.

To get around this problem, the creators of NSIS - who provide us with the software for making installation sets - were kind enough to take advantage of some features in 64bit Windows that allows us to get around redirection.  Mostly.

Unfortunately, whenever redirection is disabled, it only changes file redirection - not registry redirection.  Since our installation set is writing directly to the registry, disabling file redirection doesn't help us.  So, we have to find a way to write to the 64bit registry through file redirection.  This leads us to reg.exe - a nifty little utility that came with Windows XP that allows the registry to be accessed from the command line.  Since 64bit Windows has a 32bit reg.exe and a 64bit reg.exe, disabling file redirection should allow us to call the 64bit copy directly, which doesn't have that pesky 32bit registry limitation.

All we need to do is to check for 64bit Windows, disable file redirect, run reg.exe and then reenable file redirection. That gives us code that looks something like this, which actually does work*.

var /GLOBAL NSISRegPath
StrCpy $NSISRegPath "SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

var /GLOBAL EXERegPath
StrCpy $EXERegPath "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

${IF} ${AtLeastWinXP}
    WriteRegStr HKLM "$NSISRegPath" "$INSTDIR\program.exe" "DisableNXShowUI"
    ${IF} ${RunningX64}
        ${DisableX64FSRedirection}
        ExecWait '$SYSDIR\reg.exe add "$EXERegPath" /v "$INSTDIR\program.exe" /d "DisableNXShowUI"'
        ${EnableX64FSRedirection}
    ${ENDIF}
${ENDIF}

Finally! It works!  Total time of this endeavor is a little over an hour.  Now I'm off for more compatibility testing.  Windows 8 and Windows 2012 up next!