Configuring DEP on Windows Server 2008 R2 from a 32bit NSIS Installer - Revisited

Thanks to another blog, The Old New Thing, I found out there is another way to handle writing to the 64bit Registry from a 32bit program - without the needing to deal with file redirection.  Well, almost without the need for it. Starting with Windows Vista, a special alias %windir%\Sysnative was added to allow access to the System32 directory even when running a 32bit program.

Unfortunately, Windows XP and Windows Server 2003 don't have this feature; and since we still work with both of these versions of Windows, I cannot incorporate it into our code.  Anyway, since the original post is one of my more popular, I thought I would update the code.

var /GLOBAL NSISRegPath
StrCpy $NSISRegPath "SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

var /GLOBAL EXERegPath
StrCpy $EXERegPath "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

${IF} ${AtLeastWinVista}
    WriteRegStr HKLM "$NSISRegPath" "$INSTDIR\program.exe" "DisableNXShowUI"
    ${IF} ${RunningX64}
        ExecWait '$WINDIR\SysNative\reg.exe add "$EXERegPath" /v \
            "$INSTDIR\program.exe" /d "DisableNXShowUI"'
    ${ENDIF}
${ENDIF}

Note: Lines that end with a backslash represent long lines that have been wrapped.

Base64 Encoding using MSXML

At the time we were developing the base64 encoder - covered in my last post, there already existed a quick and dirty way to perform base64 encoding/decoding through MSXML.  Even though we have decided not to pursue this for any Platypus development, it might benefit someone.  Here are examples for using MSXML for base64 encoding/decoding from within VBScript.

    Function Base64Encode (strData)
        Set objDocument = CreateObject("MSXML2.DOMDocument")
        Set objNode = objDocument.createElement("document")
        objNode.dataType = "bin.base64"
        objNode.nodeTypedValue = strData
        Base64Encode = objNode.text
    End Function
    
    Function Base64Decode (strData)
        Set objDocument = CreateObject("MSXML2.DOMDocument")
        Set objNode = objDocument.createElement("document")
        objNode.dataType = "bin.base64"
        objNode.text = strData
        Base64Decode = objNode.nodeTypedValue
    End Function


This is covered in more detail in Microsoft's Knowledge Base Article named "How To Create XML Documents with Binary Data in Visual Basic", along with examples for hex encoding and date encoding  (ISO-8601).

Base64 Encoding in Platypus

Base64 encoding was originally designed for transmitting binary data over SMTP.  When email servers were first created, the SMTP protocol restricted the allowed characters to the first 7bits of ASCII's 8bit character set. Among other things, this prevented binary files from being sent over email.

Then came MIME and with it, base64 encoding, which converted binary files into readable text suitable for sending over email. With base64, for every 3 bytes of binary data input there are 4 bytes of encoded text returned. Compared with hex (base16) encoding, the other dominant encoding mechanism at the time, base64 encoding is much more efficient in terms of storage by lowering the amount of space needed for attachments from 2:1 (for hex) to 4:3 (for base64).  Since its creation, base64 encoding has been included in a variety of internet protocols, including SMTP authentication, HTTP authentication, XML and is used in various subsystems of the Platypus Billing System.

While Visual FoxPro - the base language for the Platypus Billing System - includes features for encoding and decoding using base64 through STRCONV(), that function does not follow the line length requirement in the RFC 2045 specification, which limits encoded lines to a maximum of 76 characters. For example, if data were encoded into 100 characters, two lines of encoded data would result.  The first line 76 characters, followed by CRLF and then the remaining 24 characters.

At first, this limitation was not a problem.  We used 3rd party ActiveX/COM libraries, named EncodeX and SmtpX from Mabry Software, for encoding email attachments and sending emails, respectively.  The other use of base64 within Platypus - our own attachment feature first included in Platypus v3 - did not have to interact with any external systems, so RFC 2045 compliance was not a requirement.

Eventually, we began work on creating a shared library that could wrap all our SMTP functionality for Platypus v5.  Up until this point, there were separate classes and libraries for the Platypus client and API.  For one, the Platypus client used ActiveX forms of the libraries, while the Platypus API used the COM forms.  Of course, this made maintaining that code doubly difficult and it was prone to inconsistent behavior between the Platypus client and API.  Because of that inconsistency, limitations in Visual FoxPro, and compatibility problems, we dropped the ActiveX form of SmtpX and completely dropped the EncodeX libraries.

Unfortunately, because FoxPro's STRCONV() function did not strictly follow MIME, sending attachments encoded using this function would often generate errors.  The strange thing was that these errors were not universal across all mail servers.  Some were more strict than others.  Anyway, to move forward, we decided to develop our own library for base64 encoding.

This was my first C++ project with Platypus.  Up until this point, my development experience - excluding school - was limited to Visual FoxPro, Visual Basic, and SQL.  After an excessive amount of research, we found a decent example in the public domain which performed encoding quickly enough and adapted it into a COM library using Visual C++ 6.0.  That library has been in use since that time for a majority of Platypus v4 and all of Platypus v5 and v6.  Not bad for a piece of of code 10 years old.  With the release of Platypus v7, that COM library has been phased out in favor of the Mailbee SMTP COM library, which handles encoding internally; but the old base64 COM library is still included in our installation sets as part of a fallback feature.

The original C++ source for the encoder has been phased out of our other components, as well; being replaced with a much improved library.  This new library takes advantage of Microsoft's SecureCRT (secure C runtime) guidelines to prevent buffer overflows, now includes an efficient decoder, and has been fuzz tested to ensure stability.  It is currently in use within the updated Mailpopper for re-encoding email attachments that have been decoded by the Mailbee POP3 COM library, which pulls emails into the Helpdesk features in Platypus.

Configuring DEP on Windows Server 2008 R2 from a 32bit NSIS Installer

If the title is hard to understand, let me just shorten it to this.  The woes of compatibility testing!

Included with the Platypus Billing System is a number of 3rd party ActiveX libraries.  Most of the time, these libraries are wondrous things.  Unfortunately, one, in particular, has a problem with Windows DEP.

Now, upon installation of the Platypus client, we can get around this by configuring the Application Compatibility settings of our executable to bypass DEP.  So, whenever the exe is launched, the OS will not trap DEP problems for our process.  We do all of this by simply writing to the appropriate location in the registry during the installation process.

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers

All the way from Windows XP up to Windows Server 2008, we simply had to write to the registry.  It didn't even matter whether the OS is x86 or x64.  It has just worked.

Anyway, one of the more ingenious and infamous features from Windows Vista is redirection.  You see, whenever a 32bit application writes to HKEY_LOCAL_MACHINE in the registry on 64bit Windows, it is actually writing to HKEY_LOCAL_MACHINE\Software\Wow6432Node. So, whenever our software was installed, because the installation set is a 32bit application itself, when it writes to the registry, it is actually writing to the 32bit subset of the registry.  This includes those pesky Application Compatibility settings I mentioned.  Up until the latest version of Windows Server, the OS didn't care whether you configure the compatibility settings in the 32bit or 64bit registry.  It checked both when the process started.  Here's an example of code we used for configuring DEP*.

!include LogicLib.nsh
!include WinVer.nsh

var /GLOBAL NSISRegPath
StrCpy $NSISRegPath "SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

${IF} ${AtLeastWinXP}
    WriteRegStr HKLM "$NSISRegPath" "$INSTDIR\program.exe" "DisableNXShowUI"
${ENDIF}

Starting with Windows Server 2008 R2, Application Compatibility settings in the 32bit registry now appear to be ignored.  Meaning the Platypus client will now crash whenever our troublesome ActiveX library rears its head, because our installation set can't get to the 64bit registry.  Not easily, anyway.

To get around this problem, the creators of NSIS - who provide us with the software for making installation sets - were kind enough to take advantage of some features in 64bit Windows that allows us to get around redirection.  Mostly.

Unfortunately, whenever redirection is disabled, it only changes file redirection - not registry redirection.  Since our installation set is writing directly to the registry, disabling file redirection doesn't help us.  So, we have to find a way to write to the 64bit registry through file redirection.  This leads us to reg.exe - a nifty little utility that came with Windows XP that allows the registry to be accessed from the command line.  Since 64bit Windows has a 32bit reg.exe and a 64bit reg.exe, disabling file redirection should allow us to call the 64bit copy directly, which doesn't have that pesky 32bit registry limitation.

All we need to do is to check for 64bit Windows, disable file redirect, run reg.exe and then reenable file redirection. That gives us code that looks something like this, which actually does work*.

var /GLOBAL NSISRegPath
StrCpy $NSISRegPath "SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

var /GLOBAL EXERegPath
StrCpy $EXERegPath "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers"

${IF} ${AtLeastWinXP}
    WriteRegStr HKLM "$NSISRegPath" "$INSTDIR\program.exe" "DisableNXShowUI"
    ${IF} ${RunningX64}
        ${DisableX64FSRedirection}
        ExecWait '$SYSDIR\reg.exe add "$EXERegPath" /v "$INSTDIR\program.exe" /d "DisableNXShowUI"'
        ${EnableX64FSRedirection}
    ${ENDIF}
${ENDIF}

Finally! It works!  Total time of this endeavor is a little over an hour.  Now I'm off for more compatibility testing.  Windows 8 and Windows 2012 up next!

Application Manifests & Visual Basic 6

Embedding Application Manifests in Visual Basic 6 binaries is really easy.  Microsoft even wrote a command line utility just for this purpose.  Well, not for VB6 but for binaries in general.  The Manifest Tool (mt.exe), which is included in both Visual Studio and the Windows SDK is extremely simple to use.  The best part is that it handles any necessary padding and can update just about any binary with no fuss.

Here's an example command line using the naming convention of Visual Studio 2005, where the manifest filename contains the program name with ".intermediate.manifest" appended. 

mt.exe -nologo -manifest "program.exe.intermediate.manifest" -outputresource:"program.exe;#1

And now for a story...

When we were first confronted with the need for manifests - specifically for triggering UAC prompts in our configuration tools written in VB6 - I performed my due diligence.  I researched the topic thoroughly, I took examples of the manifests provided by Microsoft, and I tested on each and every Windows OS we supported - Windows 2000 all the way up to Windows Vista/2008.

The one thing I couldn't find was a simple way to embed the manifest in those executables, that could be easily automated.  The articles I read covered GUI tools like XN Resource Editor and Resource Hacker, writing my own C/++ program using UpdateResource, a long winding route using the Resource Compiler (rc.exe) or finally just leaving the manifest as a separate file.

Even though manifests had been around since Windows XP, there wasn't a single article I could find that even mentioned the Manifest Tool. Even in the Microsoft articles I have found, there is never any mention of VB6 and the Manifest Tool together.  Of course, VB6 was considered legacy by the time Application Manifests came out; so, while frustrating, I can't really blame them.  I can blame my search engine skills, but that's no fun.

Anyway, all but the last option were complicated, convoluted, or required too much effort.  We, of course, finally settled on that last option - using external manifests - out of necessity to get something out the door.  It wasn't until we started migrating code from Visual C++ 6.0 over to Visual Studio 2005 that I noticed the mt.exe command line in the build log, which was over a year and half later.  Now, along with code signing, through signtool.exe the Manifest Tool is included in much of our automated build process, and I am much happier for discovering it.

Application Manifests, UAC & Windows Vista - Part 2

Way back in the days of Windows 2000, new guidelines were introduced to improve the overall usability and security of the OS.  Although, in order to maintain compatibility, none of these guidelines were actually enforced at the time.  Then, when Windows Vista was released, the game changed.  Those guidelines were no longer optional, but there was a backup plan.

New Guidelines for Windows 2000

1. Only binaries or read only files should be stored in Program Files.
2. Any documents or user created files should be stored in My Documents.
3. Temporary files should be created in the user (or system) Temp folder.
4. Anything else should be written to Application Data.

Now, in a perfect world, everyone would have read those guidelines and started following them.  After all, there is no shortage of documentation on how to implement them, but things rarely go according to plan.  Even we were not immune to this thinking.  I guess it could be summed up as "Well, if Microsoft doesn't care if we do it, then why should we change?"  And, you can't exactly cry foul just because you didn't read the rules, can you?

With the introduction of UAC, it was no longer possible to write to Program Files without administrative privileges, but somehow older programs still worked.  This is where the the requestedExecutionLevel flag in Application Manifests comes into play.  If the flag is missing from the manifest, the OS does its fancy footwork of guessing whether the program should be elevated.  If the program is not elevated, then the program is run in compatibility mode.  Any attempts to write to a system folder - such as Program Files or System32 - or write to a system registry hive - such as HKLM - will result in virtualization.

While you may think your program is writing to HKLM, it's not.  Not really.  It may look like it to you.  Even to the program itself, it will appear that way, because it is actually reading from a virtualized section of the registry.  This means that in your own little world, everything is working as you expected, but it is not directly affecting the OS.  So, any changes you make to your little world don't affect anyone else who logs into the OS.  This is one of the key points that makes UAC really work.  Without this, Vista would truly be the nightmare that you hear in those Mac commercials.

Still, even with virtualization in place, some wacky things can occur.  for example, there's a specific case I found with the SDelete utility provided by SysInternals, which runs in compatibility/virtualization mode.  If the program is run as unpriviledged and is passed a file located in Program Files, it simultaneously finds the file and cannot find the file.  It actually finds the file in Program Files, and then attempts to open a file of the same name in the Virtual Store.  So, after all that, it reports a success saying the file was correctly wiped and deleted, but the file in Program Files is never touched.

User Account Control (UAC)

• UAC Architecture 
• Designing UAC Applications for Windows Vista 
• New UAC Technologies for Windows Vista 
• Common file and registry virtualization issues in Windows Vista or in Windows 7 
• UAC - What. How. Why. 

Application Manifests, UAC & Windows Vista - Part 1

Support for Manifest Resources continued with Windows Vista.  In addition to the new Shell Common Controls library and Side-by-side assemblies introduced in Windows XP, support for High-DPI Applications and User Account Control (UAC) was added to the Manifest Resource.  The most scrutinized feature of the two, of course, is User Account Control, which provided additional security to the operating system by specifying within the program what type of permission was required for using the program.

With Windows XP and lower, an administrator was always logged in as an administrator and a restricted user was always logged in as a restricted user.  This meant that any program executed by an administrator would always be executed with full administrative rights.  While any program executed by a restricted user would never have administrative privileges.

With the introduction of UAC, if a program required administrative privileges, such as writing to HKLM in the registry or writing to a system folder, the OS would notify the user of the required privileges before executing potentially damaging code - even when logged in as an administrative user.  The OS determined this through the requestedExecutionLevel flag in the application manifest.  If this flag was missing, the OS would do some fancy footwork and make a guess.

The key point here is that by embedding the required privilege within the program, a user - even a user with administrative privileges - did not have to work in a completely unprotected environment.  This helps restrict access to much of the operating system when performing day-to-day tasks; and when malicious software is introduced on the machine, the restricted access given to that software should help mitigate any damage.

Vista Styles

• Microsoft Download - Windows Vista/7 User Interface Guidelines 
• MSDN Library - How to Create the Best User Experience for Your Application 

User Account Control (UAC)

• MSDN Library - The Windows Vista and Windows Server 2008 Developer Story: Windows Vista Application Development Requirements for User Account Control (UAC) 
• MSDN Library - Windows Vista Application Development Requirements for User Account Control Compatibility 
• MSDN Library - Designing UAC Applications for Windows Vista - Step 6: Create and Embed an Application Manifest (UAC) 
• Microsoft TechNet - User Account Control Step-by-Step Guide
• Microsoft TechNet - Understanding and Configuring User Account Control in Windows Vista

Compatibility

• The App Compat Guy - Exploring Manifests Part 1: Side by Side Assemblies
• The App Compat Guy - Exploring Manifests Part 2: Default Namespaces and UAC Manifests in Windows Vista
• The App Compat Guy - App Compat Bug Explained: The Mysterious Black Background in Visual Basic 6 (VB6) Forms and Controls

Application Manifests & Windows XP

Starting with Windows XP, Microsoft created extensive changes to the Windows Manager and Shell Common Controls. These changes included a whole new set of controls that could be used with programs to provide a fresh look and to provide additional Theme support. Older programs would continue to use the older Shell Common Controls library (comctl32.dll), while newer programs would use the new Shell Common Controls library (uxctrl.dll). By having two separate libraries instead of updating the existing Shell Common Controls library, compatibility issues with older programs were completely bypassed.[1].

To determine whether programs would use the new library was accomplished through a new PE Resource type called a Manifest. This Manifest Resource - in its original form - contained a list of DLL's and other resources that would be used with the executable. This allowed easier control over dependencies by specifying which versions of a DLL would be used with the program - specifically which Shell Common Controls library. So, newer programs, which included a specific Manifest Resource, would use the uxctrl.dll; and older ones, which did not have a Manifest Resource, would continue to use comctl32.dll.

This created the concept of Side-by-side Assemblies - a new feature in Windows XP - where multiple versions of a DLL could be installed on a machine, and depending on the Manifest Resource within the binary only a specific version of that DLL would be used by the program. For several reasons Microsoft has shifted away from supporting side-by-side assemblies - at least in C/C++ runtime libraries - in favor of including the version of the appended to the file name.[2]

XP Styles

• Microsoft Download - Windows 2000/XP User Interface Guidelines
• The Old New Thing - The history of the Windows XP common controls
• MSDN Library - Using Windows XP Visual Styles With Controls on Windows Forms

Side-By-Side Assemblies

• MSDN Library - Side-by-side Assemblies
• MSDN Library - Dynamic-Link Library Search Order
• Wikipedia - Side-by-side Assembly
• MSDN Library - Using Side-by-Side Assemblies as a Resource
• MSDN Library - Enabling an Assembly in an Application Hosting a DLL, Extension, or Control Panel
• MSDN Library - Enabling an Assembly in an Application Without Extensions
• MSDN Library - Per-application Configuration on Windows Server 2003
• MSDN Library - Per-application Configuration on Windows XP
• MSDN Library - Troubleshooting C/C++ Isolated Applications and Side-by-side Assemblies
• MSDN Library - Concepts of Isolated Applications and Side-by-side Assemblies
• Microsoft Support - A new CWDIllegalInDllSearch registry entry is available to control the DLL search path algorithm
• Microsoft Support - Some third-party applications that use external manifest files stop working after you install Windows server 2003 Service Pack 1
• MSDN Library - Dynamic-Link Library Redirection
• Nothing ventured, nothing gained - DLLs and resource ID 2 manifests