Upgrade to VMWare VI 5.5 going wrong?

When upgrading to VI 5.5 all seemed to go fine, until Virtual Machines started failing to vMotion. Failing at 13% with the wonderfully clear and helpful explanation: "a general system error occurred: invalid argument"

Further to that, our Veeam installation was failing to replicate and backup with the following errors:

Error: Client error: NFC storage connection is unavailable. Storage: [stg:datastore-446,nfchost:host-6879,conn:]. Storage display name: []. Failed to create NFC download stream. NFC path: [nfc://conn:,nfchost:host-6879,stg:datastore-446@/.vmx]. 

A similar error was produced if you tried to browse the datastores and download a file from them.

None of the solutions on Veaam or VMWare helped. 

The only solution was to rebuild the Host from scratch using a 5.5 image. Of course, without vMotion live VMs cannot be migrated. But rather than shutting down and cold migrating all the VMs, just suspend them. It's only Live Migration using vMotion that fails. VMs are then only down for ~30 seconds if you are quick.

Xubuntu 12.10 on VMware Desktop Crash

Using Xubuntu (server edition) as a CUPS print server serving AirPrint clients seemed like a good idea and, on the whole has worked out to a fair degree of success. We did have one problem that could have been a show stopper.

When logging out of the X session, the lightdm login manager would crash without recovery. The same thing happens if you try to change the desktop resolution from its initial cramped 800x600 workspace. These days, anything smaller than 1024x768 feels like the dark ages.

The solution? Create your own config. Now X used to create xorg.conf files based on user input. These days X does everything on the fly based on auto-detect. The xorg.conf is therefore defunct and on its way out the door. You can always try to run X -configure (remembering to shutdown your existing xwindows session) but you might run into the same problem as us, where the configure script runs into problems and panics, leaving you without a configurable script.

That is not the end of the World. The brave new world of the modern X server now uses additional files that it finds in the /usr/share/X11/xorg.conf.d directory. These additions are added to the auto-config on startup and prove to be very useful in stabilising what might end up being an unstable configuration.

We are running VMware 5.1. The VM does not have 3D graphics enabled. We created a file in said directory called 10-monitor.conf and added the following information:

Section "Monitor"
    Identifier    "Monitor0"
    HorizSync 1.0 - 10000.0
    VertRefresh 1.0 - 10000.0
EndSection
Section "Device"
    Identifier    "Device0"
    Driver        "vmware"
EndSection
Section "Screen"
    Identifier    "Screen0" 
    Device        "Device0"
    Monitor       "Monitor0"
    DefaultDepth  24 #Choose the depth (16||24)
    SubSection "Display"
        Viewport 0 0
        Depth       24
        Modes     "1024x768"
    EndSubSection
EndSection

Save the file and log out. Now the login manager displays at 1024x768 and after logging in, changing the resolution to 1024x768 no longer produces the hang and crash scenario.

RPC over HTTP with Apache's mod_proxy

Well, let us get the bad news over and done with. Unless you revert to 2.0.53 or earlier, you can forget proxying RPC over HTTP with Apache web server.

Why is this?

Apache prefetches the entire body before establishing a connection. This mitigates a serious smuggling exploit that is detailed here: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-2088

Microsoft's idiosyncratic implementation of RPC over HTTP establishes two connections, RPC_DATA_IN and RPC_DATA_OUT, in order to send data between the Outlook Client and the Exchange Server, or between the Remote Desktop Client and Desktop Gateway. The problem is that the client sends a minuscule amount of data over the RPC_DATA_IN connection, somewhere in the region of 100 bytes. This is not enough to trigger a connection in the Apache implementation of HTTP/1.1

While not strictly forbidden in the RFC, it does mean that allowing the connection creates an insecure environment. Consequently no amount of begging is going to make Apache developers change their minds. To make matters worse, Microsoft in their protocol publication, have applied their right to enforce patents on the use of their protocol. Game over.

So if you are looking for RPC over HTTP, you will have to stay with Microsoft, who, in their infinite wisdom, have apparently ditched their very successful ISA/TMG product without a replacement. Either that or not use Microsoft products in future. We may well tell our users that Outlook Anywhere is no longer an option rather than punch holes in our firewalls to allow what is by design an insecure protocol.

Building Apache 2.4.4 from the Unix source with Openssl, zlib, libxml2, ipv6 support on Windows with Visual Studio 2012 to x86 binaries

Why would I want to even think about it? Well, because all the binaries compiled for windows have IPv6 disabled by default and the only way to enable it is to compile it yourself. Now I have to warn you, it took me well over a week to figure out what was going on and had endless failures. This procedure that follows is likely to test your patience, however, the result is a working Apache httpd running on Windows Server 2012.

That is, one that has been compiled from the original sources, and not one that has been added to by Bob down the road who has slipped in his own datalogger, if you know what I mean...

The main requirements are: Visual Studio 2012, the Platform latest SDK, perl and awk in the environment path, and CMake. Next...

Download it all

Download the latest version of apache source and unpack (Unix sources suffice) into a httpdxx root directory. All the following packages will end up in the srclib directory.
Download latest version of openssl source and unpack in to sub directory srclib\openssl
Download latest version of zlib source and unpack into in to subdirectory srclib\zlib
Download latest pcre source
Download the expat binaries compiled for windows and put into srclib\win32\bin\release
Download the prceombiled binaries and the dev packages for libiconv and copy into srclib\libiconv
Download the source for libxml2 and place into srclib\libxml

Build pcre

Unzip source into a seperate directory from your httpdxx directory. Add a build subirectory as a destination for your CMake build. Use CMake to create a solution for native Visual C 11 with the following options:
• BUILD_SHARED_LIBS Yes
• CMAKE_INSTALL_PREFIX (your httpdxx\srclib\pcre folder)
• PCRE_SUPPORT_UTF
Open the solution in Visual Studio and change to Release. Check the project to make sure that PDB’s are created. Compile the BUILDALL project. If all compiles, build the INSTALL project. Make sure the binaries end up in the same place.
Copy the pcre.dll and pcre.pdb into the srclib\pcre (needed because of badly written includes and this was the easiest option.)

Build zlib

Open your Visual Studio Dev Environment command line and cd to httpdxx\srclib\zlib
open VS Command Prompt and navigate to srrclib\zlib. build using:
nmake /f win32\Makefile.msc
nmake /f win32\Makefile.msc test

Build Openssl

cd to httpdxx\srclib\openssl
perl Configure VC-WIN32 enable-camellia no-idea no-rc5
(You do not really want zlib compiled into openssl do you? I’m shocked. Really!)
ms\do_ms.bat
nmake -f ms\ntdll.mak
Create an empty file store.h in srclib\openssl\inc32\openssl or OpenSSL will fail to build into the rest of the projects.

Prepare httpd

Update srclib\apr\include\apr.hw and change: (default is 0) This enables IPv6
#define APR_HAVE_IPV6           1
Run the following from the root of the httpd source to deal with UNIX line endings and prepared the projects for use with .net compilers:
>Perl .\srclib\apr\build\lineends.pl
>Perl .\srclib\apr\build\cvtdsp.pl -2005
Now open Visual Studio and open the Apache.dsw workspace file. It will convert the file to a Apache.sln and convert all the projects to .vcxproj.
Change properties of libhttpd:
Configuration Properties -> Linker -> General -> Additional Library Directories add .\srclib\pcre\lib (Location of pcre.lib)
Configuration Properties -> C/C++ -> General -> Additional Include Directories add ./srclib/pcre/include (location of pcre.h)
Open ApacheMonitor.rc and comment out the following line with a #:
"CREATEPROCESS_MANIFEST_RESOURCE_IF RT_MANIFEST "ApacheMonitor.manifest""
Change the Properties of ApacheMonitor:
Configuration Properties -> C/C++ -> Manifest Tool -> Input and Output -> Additional Manifest Files. Add ApacheMonitor.manifest
Change the properties of Mod_xml2enc
Configuration Properties -> C/C++ -> General -> Additional Include Directories add ./srclib/apr-iconv/lib

Build Libxml2

The Libxml2 source contains a folder called win32\VC10. Add the libxml2.vcxproj to your Apache solution. Change the properties of the libxml2 project. (You may need to upgrade the project first by going to Project -> Upgrade Projects.)
Configuration Properties -> General
-Configuration Type Dynamic Library (.dll)
Configuration Properties -> C/C++ -> All Options
-Enable Function-Level Linking: No
-Enable Instrisic Function: No
-Output Directory ..\Release (Or it will go into the httpd\Release folder. Not a good idea. You will need to create a win32\Release folder)
Configuration Properties -> Linker -> All Options
-Output File: ../Release/libxml2.dll
-Additional Library Directories: ../../../libiconv/lib;../../../libiconv/bin
-SubSystem: Windows
-Additional Dependencies: libiconv.lib;libcharset.lib;ws2_32.lib;mswsock.lib;
-Link Time Code Generation: Default
-Generate Debug Info: Yes
Modify mod_proy_html and mo_xml2enc. These two projects need Lib2xml.
Configuration Properties -> C/C++ -> All Options
-Additional Include Directories: ../../include;../../srclib/apr/include;../../srclib/libiconv/include;../../srclib/apr-iconv/include;../../srclib/apr-util/include;../../srclib/libxml2/include;
Configuration Properties -> Linker -> All Options
-Additional Library Directories: ../../srclib/libxml2/win32/Release;../../srclib/libiconv/lib;../../srclib/libxml2/win32/bin.msvc;
-Additional Depdencies: (Add) ws2_32.lib;mswsock.lib;

Build Apache

Go to Solution Congifuration. Change to Release and remove the build tick from the following projects:
• all projects beginning apr_db
• BuildAll
• BuildBin
• Fcgistarter
• gen_test_char
• InstallBin
• Libarpiconv_ccs_modules
• Libarpiconv_ces_modules
• Mod_lbmethod_rr
• Mod_lua
• Mod_session_crypto
• Mod_socache_dc
Build the Solution. This should allow everything to build successfully. There may well be something I have missed. Believe me, it is possible to fix it! I did.
Open your Visual Studio Dev Environment command line.
>Cd \htppdxx\srclib\apr-iconv\ces
Use NMake to create a batch file from the MakeFile instructions:
>NMAKE /nologo /f Makefile.win BUILD_MODE="Win32 Release" BIND_MODE=shared /N /U >>make.bat
Edit the make.bat file to remove all the leading tabs and remove all of " /Yuinconv.h " This stops the building of components using precompiled headers, otherwise you get an external reference error.
Run the batch file to compile these, do NOT compile them in Visual Studio.
Do the same in the ccs directory.
Amend Makefile.win. Find the section list below and add the line in bold:
!IF EXIST("srclib\libxml2")
 copy srclib\libxml2\win32\$(LONG)\libxml2.$(src_dll) "$(inst_dll)" <.y 
 copy modules\filters\$(LONG)\mod_proxy_html.$(src_so)  "$(inst_so)" <.y
 copy modules\filters\$(LONG)\mod_xml2enc.$(src_so)  "$(inst_so)" <.y
Save MakeFile.win.
Finally go to the root of the source directory and run the install script to pull out all the compiled files into an installation directory (in my case D:\Apache24)
>cd \httpdxx
>nmake -f makefile.win INSTDIR=D:\apache24 SHORT=R LONG=Release _install

Strip the Binaries

You now have about 86MB of Apache Server. You can make that a lot lighter by doing the following.
Remove all the debug information (i.e. del /S /Q *.pdb)
Delete the Manual Directory (That why Google was invented)
Delete the Include and Lib directories (Unless you are going to do some development work using this set of binaries.)
You should now have about 8MB of web server.

Install on Server 2008 R2/2012

Install the Visual C redistributable (You will need the 2012 version, of course)
Copy your cut down binaries to your chosen location on your server.
From a (elevated) Command Prompt, navigate to bin directory and run:
httpd.exe –k install (installs as a service and starts…)
Configure Server using the configuration files
Now – I cheated – of sorts. In order to get an initially working conf file, I installed apache using the MSI and ran through the initial configuration. When I then compiled my own version of apache, I used the same working configuration file and amended it from there.

Configuration Gotchas

SSLPassPhraseDialog The default setting (builtin) does not work on windows so you need to point to a pass phrase programme. I used Notepad.exe. I removed the pass phrases from my passwords (Just Google “retrieving a key and certificate from a PFX file.” Very effective)
SSLSessionCache This requires a path not broken by spaces. If you have installed to the default location (as with the msi installed provided by apache) then you’re not quite stuck, but it feels it. Create a shortcut in the root of the c drive that points to your apache directory. Then use shmcb:C:\Apache\etc…

And that is it. As I have said, I have probably missed some things out here, but I do have a working solution, so it is possible, if akin to having your fingernail being pulled out.

This year's IPv6 Roadblock

Are you wondering why most companies are not looking at IPv6? Why is adoption so slow?

The answer may be Web Filtering!!!

Go back 5 years and the blocks to IPv6 adoption were multiple. Device compatibility, especially firewalls and switching, applications. Even Active Directory in its 2003 form could not support IPv6.

That is all past. Today, lack of IPv6 support is a warning sign that the company you are purchasing from is lazy on the development front. Consequently, such companies are on a slow downward trend. Watch carefully and you'll see them disappear or bought out. Except for Web Filtering.

Most filtering companies are still on the edge of the cliff, gathering together for warmth and seeing who is going to jump in first. Well guess what, no one with any mind for web security (which is just about every minor organisation with the financial oomph to purchase such a package) is going to go IPv6 for web browsing until filtering is in place. Companies that look into IPv6 adoption don't jump and then look how they are going to do things, they want the whole project plan down first. So when they get to the final step and find that they cannot cross it, the whole project fails.

So it is not that companies are not looking for IPv6 filtering, they are just waiting for Web Filtering companies to get on board. Guess what, Web filtering companies are waiting for firms to tell them they want it. That is not going to happen! So if any web filtering company is reading this, want to get ahead of the competition in this tough season, IPv6 is your boost. Get in first and you may give yourself that needed lift to not only survive the current financial crisis, but to come out fighting.

Exchange 2010 UM and WMF 3.0

Windows Management Framework 3.0 is about to be rolled out over Windows Update. Do you know what will break? I do! Exchange 2010 Unified Messaging. Specifically the part which uploads media files to the server for use in the Auto Attendants. When you try to do this you will get a WinRM error:

Starting a command on remote server failed with the following error message : The WinRM client sent a request to the remote WS-Management service and was notified that the request size exceeded the configured MaxEnvelopeSize quota. For more information, see the about_Remote_Troubleshooting Help topic.
    + CategoryInfo          : OperationStopped: (um01.westminster.org.uk:String) [], PSRemotingTransportException
    + FullyQualifiedErrorId : JobFailure
    + PSComputerName        : um01.westminster.org.uk

This is the PowerShell version of the error. The EMC will weakly complain that it has been given the above message with roughly the same text. Can you live without the ability to upload message to Exchange? I thought not.

Trust me, the about_Remote_Troubleshooting Help topic will not help you with this one. Changing the WinRM MaxEnvelopeSizekb value will not make a difference. The release notes point out that this has been change to 700. You can find out what WinRM config is set to by opening an elevated command prompt and typing 'winrm g winrm/config' to display the current config.

But WinRM/config is not the only place that MaxEnvelopeSize is specified! (See http://msdn.microsoft.com/en-us/library/windows/desktop/ee309364(v=vs.85).aspx) Hosted applications (such as Exchange PowerShell) also have their own configuration in IIS. This is typically in the system.management.wsmanagement.config xml portion of the web.config. Exchange PowerShell module has it's own virtual directory in IIS and it is here that you will need to make an amendment in order to restore functionality back to UM.

Find the web.config associated with the PowerShell IIS Virtual Directory. Found usually in the \Client Access\PowerShell directory. Make a backup. Amend the OperationsConfiguration node to add the MaxEnvelopeSizeKB="700" setting. Replace 700 with whatever MaxEnvelopeSizekb is in WinRM. (You've probably been trying to amend the value like I did and sinking into despair when it makes little difference. Go on. Admit it.) Mine now looks like this:

 
   
      ...
      <operationsconfiguration MaxEnvelopeSizeKB="700" ... >
</operationsconfiguration><
...
      ...


Reset IIS and the error should be gone. There are, of course, other issues with WMF that are detailed elsewhere. The most significant is the fact that WinRM IIS Extensions for me needed re-installing after WMF was installed, but that is well documented elsewhere.
As an additional warning, if you want to run update rollups, you will need to uninstall WMF 3.0 first before installing the rollup, and then reinstalling WMF.

Getting SQL 2008 R2 Reporting Server working with System Center 2012 Configuration Manager

When you upgraded to System Center 2012, you discovered that in order to get reports from Configuration Manager, you now absolutely need to get Configuration Manager working with SQL Reporting Services. You follow the instructions provided by Microsoft and then you hit the train wreck!!!

Error 7403 - The report server service is not running on Reporting Service Point server "MySMS.domain.org.uk"; start the service to enable reporting.

Nothing out there seems to fix the problem. I have discovered that there are a number of problems that cause this error, hence the one or two 'fixes' out there are all red herrings. There are, of course, clues. I followed them and have come to the following solution to the problem.

What's the problem?
The problem occurs mainly when you have the SQL Server, the SQL Reporting Services on the same server as the Management Point. This means also that IIS is installed. That is not a problem, mind you. Reporting Services does work happily with IIS, but as you will see, there are caveats. After following all the installation you find that SMS_SRS_REPORTING_POINT under Component Status, has a Red X instead of a Green Tick. List the error events you get a repeated performance from Error 4703. It is also possible that if you go to Reporting Service Configuration Manager, you can sometimes get the ReportServer website to come up, and sometimes not, but only on HTTP, and not on HTTPS. You also see no reports and any attempt to create a report brings up AUEM (Another Useless Error Message.) Checking the srsrp.log you notice the error: 'The underlying connection was close: Unable to establish a secure connection.'

Problem 1 - The certificate Subject
If you have followed Microsoft's installation plan to the letter you probably have an IIS certificate. The subject on the certificate will be the host name of the Management Point (e.g. 'MySMS'.) This is fine, perfect for Configuration Manager. However, I bet when specify the Site Server, you specify them using their FQDN (e.g. 'MySMS.domain.org.uk'.) Can you guess what I am about to say now? Configuration Manager now cannot connect to SRS because the certificate is invalid! As I said - train wreck!

Solution 1 - The first thing you need to do is uninstall the Reporting Point, as you will need to re-install it later. Get yourself a new certificate from your CA with the FQDN as the subject and a uniquely identifying name (I used "For Report Server") as the subject and install it in the computer's personal certificate store.

Problem 2 - Conflicting Bindings
But it's not a simple as that. Why? Because Reporting Services shares the TCP Port with IIS. IIS needs to be using the SMS IIS Certificate. Despite the fact that you can specify a separate certificate in IIS to that in Reporting Services, they both present their Certificate to the OS to handle. The result is that only one certificate can remain on that particular binding (IP and Port combination.) In the report server log you see 'Failed to register url=https://MySMS.domain.org.uk:443/ReportServer...' etc.

Solution 2 - Change the Binding in Reporting Services Configuration Manager (RSCM.) Now, funny enough guess what? You might have another problem.

Problem 3 - Lingering URLs in Reporting Services
When you change the certificate and binding in RSCM you sometimes find that although you have changed the binding, the original link to 443 remains. If it does, Reporting Services will still fail to bind everything.

Solution 3 - Remove all bindings first and stop the Reporting Services Service. Open \Program Files\Microsoft SQL Server\MSRS10_50.\Reporting Services\ReportServer\rsreportserver.config (remembering to take a backup first!!!) and remove any lingering URL Reservations. You will find them under ... For me it was okay to remove the entirety and leave a behind, if you understand xml. If not, just delete everything in between the individual  and . Restart the Reporting Services Service.

Wrapping it all up
Before you re-configure your bindings in RSCM I would look at netstat first. Using the command line 'netstat -ano -p TCP' list out all the bindings. I did not do this and guess what, I picked another port that was already in use! I would suggest using a custom HTTP port as well as a custom HTTPS port, for the simple reason that you will see it being bound in netstat. It's okay to leave it and share the binding with IIS, but it does mean that you won't see the successful binding.

Okay - you have picked your ports and you go back to RSCM. Go to the "Web Service URL" tab and click on Advanced. Reporting Services automatically picks a host only URL if you enter an IP Address. You don't want this, for sanity's sake. Add the HTTP identity using your custom port (I used 81) and the FQDN for the host header. Add your SSL Binding selecting (All IPv4), add you custom currently unused port (I used 446 - 443 was used by IIS and something else had 445) and select your uniquely named certificate (with the "For Report Server" name.) Do the same for IPv6 (You are using IPv6 - >frown<) and confirm. Do the same for the "Report Manager URL" tab. Now go back to netstat and note that you now have new connections (for me on 81 and 446.)

Great. Now Navigate to your ReportServer url (launch Internet Explorer using 'Run As Administrator')  on SSL and note that you do not get any certificate warnings. Start to feel happy.

Install the Reporting Point once more. You do not have to worry about port numbers, Configuration Manager figures this out. You will need to wait about ten minutes (should be less) and then check your reporting service URL again. Configuration manager will have added a folder. Go to the Monitoring Tab of Configuration Manager and check out the large number of default report that have suddenly appeared. Finally, notice that you now have a green tick next to your SMS_SRS_REPORTING_POINT component status.

Go back to you boss and state to him that you single handly resolved the problem that had been bothering him for the last six months. Brownie points all round!!!