Debug Xen Hosted Windows Kernel Over Network

June 11, 2009

Blue screens are not a rare commodity when working with virtualization. Most of the times, full crash dumps do the trick, but sometimes live kernel debugging is required. Hard disk related crashes that prevent memory dumping is a good example where it is required, but there are times where it’s just easier to follow the entire crash flow instead of just witnessing the final state.

Type 2 (hosted) virtualization usually comes with an easy solution. But type 1 (bare metal) virtualization, like Xen, complicates matters. Debugging must be offloaded to a remote Windows machine. The common solution, it seems, is to tunnel the hosted machine’s serial connection over TCP to another Windows machine where WinDBG is running, waiting anxiously for a bug check. There are many websites describing this setup in various component combinations. I have gathered here all the tricks I could find plus some more of my own to streamline the process and get rid of commercial software.

Lets dive into the nitty gritty little details, shall we?

Hosted Windows

Kernel debugging requires some boot parameters. Windows XP includes a utility called bootcfg.exe that makes this easy.

bootcfg /copy /id 1 /d "kernel debug"
bootcfg /raw "/DEBUG /DEBUGPORT=COM1" /id 2 /a
bootcfg /raw "/BAUDRATE=115200" /id 2 /a
bootcfg /copy /id 2 /d "kernel debug w/ break"
bootcfg /raw "/BREAK" /id 3 /a

This assumes you have only one operation system configured in Windows boot loader. If the boot loader menu shows up when Windows boots, you might need to add the flags on your own to C:\boot.ini.

Xen Host

Windows will now try to access the serial port in search of a debugger. Xen’s domain configuration file can be used to forward the serial port over TCP. Locate your domain configuration file and add the following line. The configuration files are usually located under /etc/xen.

serial='tcp::4444,server,nowait'

Debugger Machine

The server side is set and it’s time to move on to the client. As previously mentioned, WinDBG doesn’t care for TCP. Instead of the usual TCP to RS-232 solution, named pipes are used here. I wrote a little application called tcp2pipe (download available on the bottom) which simply pumps data between a TCP socket and a named pipe. It takes three parameters – IP, port and named pipe path. The IP address is the address of the Xen host and the port is 4444. For named pipe path, use \\.\pipe\XYZ, where XYZ can be anything.

tcp2pipe.exe 192.168.0.5 4444 \\.\pipe\XYZ

All that is left now is to fire up WinDBG and connect it to \\.\pipe\XYZ. This can be done from the menus, or from command line.

windbg -k com:pipe,port=\\.\pipe\XYZ

To make this even simpler, you can use kdbg.bat and pass it just the IP. It assumes WinDBG.exe is installed in c:\progra~1\debugg~1. If that’s not the case, you’ll have to modify it and point it to the right path.

tcp2pipe

Source code is included in zip file under public domain.

Download tcp2pipe.zip (mirror).

Happy debugging!

1 Comment | Xen | Tagged: , , , , , | Permalink
Posted by kichik

Ouch! Spies attack the US power grid

April 10, 2009

“The power grid is becoming a bigger target for hackers as more pieces of it are connected to each other or, in some cases, to the Internet.  Employees who work remotely can be a major point of weakness. If their computers can be compromised, hackers can begin working backward into a utility’s central control system. One way that’s done is by so-called “spear phishing,” or trying to fool people into opening personalized e-mails that have malicious programs inside them. Malicious Web applications can be another route for hackers.”

A story that came across the wire yesterday detailed how spies have compromised the U.S. electric grid.  The excerpt above describes that a major security weakness is the client machine- the desktop or laptop that employees use to access their corporate network.  As we see every day among our customer base, there is a growing trend in the amount of intrusions coming into the corporate or government networks via vulnerabilities on the client computers.  The fact is that a SINGLE computer is living a “multiple personality” life, with corporate resources, patient records, credit card data, network settings, applications, and user state all intertwined in the same OS instance as browsers accessing the public Internet, Facebook apps, games, Skype, instant messaging, photo sharing websites, iTunes, etc.  This is a very dangerous practice and IT departments that permit this environment on their desktops and laptops are virtually begging for an attack.  Running all of these environments in a single OS instance is the equivalent of walking around a helium factory with a lit blowtorch.  Eventually, the two are going to combine and ignite with catastrophic consequences. 

The future of the desktop is a multi-VM environment, predicated on a thin hypervisor that dramatically reduces the attack surface for hackers.  In this environment, such as Neocleus’ upcoming “Mako” (codename) product, each VM is centrally controlled by policies set by IT and has 100% isolation and separation from the other VMs running on that machine.  So, in essence, IT can completely lock down the corporate VM (for instance, the one with applications that provide access to the electric grid), while still providing end users with other VMs that might enable a freer or more open environment.  If an intrusion is brought into the free VM, you can be 100% assured it will not infect the other VMs.  This is one of the true breakthroughs of the “Type 1″ virtualization world, as opposed to Type 2 virtualization, where in a similar scenario, would infect not only the host machine but all the guest VMs running within it. 

Leave a Comment » | Business Solutions, Government, Security, Technology | Permalink
Posted by billcorrigan

451 Group Report on Upcoming Neocleus Beta

March 7, 2009

Last Friday Rachel Chalmers of the 451 Group wrote a nice piece capturing the excitement and opportunity for Neocleus’ upcoming beta  of our Client Management & Virtualization suite, code named “Mako”, which is built upon the 2nd generation Client Hypervisor.  She describes how our platform can reduce the cost and complexity of managing PCs in the Enterprise while providing a new level of security that has not been attainable with the “one-OS-per-computer” model in place with desktops and laptops today.  Now that VT-enabled desktops and laptops can be a reality, there is a new paradigm emerging – one that we’ve seen in the data center over the past few years – where single computers will have multiple OS instances running simultaneously.  The shift to now virtualize desktops and laptops will completely transform client computing.

However, this next generation of client computing will not completely negate or require ripping out of old solutions.  Rather, an evolving ecosystem of partners where new solutions complement existing ones will emerge.  For instance, within the configuration and management arena while a new class of capabilities will be needed to manage the OS containers, existing tools will still be required to run the applications that sit inside them.  The upcoming Neocleus ”Mako”  product for example is built to work in conjunction with the existing tools such as Microsoft SCCM, Avocent / Altiris, BigFix and others.  Our centralized management and virtualization platform is designed to provide “container” management of the VMs running on the target endpoints, but not to supplant the existing management platform on these machines. OS instances that run on Neocleus operate just like native OS instances to the Change & Configuration management tools.  As a result, patching the OS, adding applications, changing settings, etc. are implemented in the same manner as without virtualization – it is just easier and more efficient with Neocleus. 

The Neocleus platform also complements VDI based solutions.  Using Neocleus, users can “check out” a VM or image from the VDI data center to take on the road with them, then “check it back in” when reconnected.  With Neocleus organizations realize increased security, centralized management and flexibility on the client while protecting and extending the investment in VDI.

We realize client virtualization will not be the only way to address the systemic desktop and laptop challenging facing IT.  However, by bringing virtualization to the endpoint, new levels of flexibility, supportability and security can finally be realized.  We’re excited that Rachel sees Neocleus as the initial driver in the industry and look forward to working with partners to deliver awesome solutions for customers.
 

 

 

Leave a Comment » | Analyst Coverage, Solutions, Uncategorized | Tagged: , , | Permalink
Posted by billcorrigan

Obama’s Helicopter and the case for SECURE CLIENT VIRTUALIZATION

March 3, 2009

Just in case you missed it amidst the mutliple bailouts, economic stimulus package, tanking stock market and other news across the world this week, there was a pretty important story about President Obama’s helicopter, Marine Corps One - and how sensitive engineering and communication information about the helicopter were compromised due to a peer to peer communication program running on a defense contractor’s computer.  Wow!  You might ask yourself, “why in the world was a P2P program running on a computer alongside critical top-secret information such as engineering specs of the newly-elected President’s helicopter?”   In a perfect world, this would never happen.  However, in the real world, we all know that the “first name” of a computer is “personal” and all users tend to personalize their laptops and desktops. 

This is a perfect example of the mission-critical need to implement technology that enables IT to lock down the corporate desktop while providing users’ the ability to personalize his/her computer.  We of course are biased, but Neocleus’ client-hosted virtualization and management platform is a breakthrough technology that hits this solution out of the park, providing IT the ability to separate corporate applications and data (such as Obama’s helicopter diagrams) from any personal applications (e.g. Skype, iTunes, Facebook, Twitter) with 100% isolation and full native OS performance.  This is just one of several use-cases for this platform, and hence why we are so excited about the upcoming solutions for customers.

Leave a Comment » | Business Solutions, Government, Security | Tagged: | Permalink
Posted by billcorrigan

Short FAQ on Desktop Virtualization

February 20, 2009

The client virtualization market is really heating up.  Here are some great statistics about the market culled from today’s Network World FAQ on desktop virtualization:

  • IDG Research Group conducted a survey of 340 IT professionals and found that 41% were investing in desktop virtualization.  They also stated that 6% of desktops were virtualized with anticipated levels hitting 1/3 of all desktops running virtually by the end of 2010
  • Gartner predicts worldwide hosted virtual desktop revenue to quadruple from $74M in 2008 to $299M in 2009

The key point is that astute IT professionals are looking for ways to reduce the overall cost of maintaining their infrastructure so that funding can be redirected toward higher-growth projects.  Forrester estimates that upwards of 60% of IT budgets are spent on operations costs associated with just keeping the status quo.  This is unacceptable, especially in today’s economic climate.  So, cost-savings alternatives such as Neocleus’ client-based virtualization solutions are ways that IT is exploring to cut costs at the desktop (while, by the way, making the desktop a lot more secure.)

Leave a Comment » | Uncategorized | Permalink
Posted by billcorrigan

Network World Article About Client Virtualization

February 13, 2009

Jon Brodkin wrote a succinct overview of Neocleus and the upcoming products on Network World today.  Jon does a great job of capturing a snapshot of the company, and the emerging market of client-based virtualization.  He also did some nice research, checking with Forrester analyst Natalie Lambert, who follows this space quite closely.   We are super excited about the future of client computing and appreciate the positive press.  As more and more customers look to reduce their overall TCO on the client, we believe there will be a huge wave of adoption of these technologies. 

When you get a chance, please check out Jon’s article here.

N.B.  Update:  Jon’s story was also picked up by Techworld, a UK-based resource for IT infrastructure information.  Glad to get the coverage on that side of the pond as well!  I am sure that many of my old friends from the Softricity partner channel will ping me after reading this.

Leave a Comment » | Public Relations, Technology, Uncategorized | Permalink
Posted by billcorrigan

It’s Late Night… with Neocleus Virtualization

February 13, 2009
("Late Show with David Letterman" Marquee at the Ed Sullivan Theater)

("Late Show with David Letterman" Marquee at the Ed Sullivan Theater)

New York – I was in NY this week meeting with a few of our customers and analysts and walked by the Ed Sullivan theatre, where David Letterman makes audiences laugh on a nightly basis. One of my favorite segments is the “Top 10” (a hysterical recent one on A-Roid is here), so it gave me an idea for a blog posting.
I haven’t seen such an enabling technology as client-hosted virtualization since the early days of application virtualization, where we would always get the “a ha” moment from customers when we showed multiple copies of Office running simultaneously on the same client. Very cool, cost-reducing technology for sure. Comparatively, the solution possibilities for Neocleus’ Type 1 client-hosted virtualization dwarf those that application virtualization provides. So I am super excited about the near and long-term for this technology. In homage to the great David Letterman, here is a quick list of just a few of the ways customers, ISVs and OEMs can solve tough IT problems using this enabling platform.

Top 10 Things You Can Do with Client Virtualization (in no particular order)

10. Reduce the cost of managing the desktop by finally realizing that elusive locked-down corporate desktop while allowing users the freedom and fidelity of using the Internet, instant messaging, VoIP, and other non-critical applications that are often the entry point for viruses, phishing tools and other intrusions.
9. Create a secure client experience by moving antivirus programs outside of the OS it is designed to protect.
8. Create a much more robust device management system by enabling critical software resources such as networking and VPN to be moved outside of the OS.
7. Reduce cost and complexity of OS migration – Easily allow the legacy OS to run side-by-side with the new OS, either on an app by app basis or for the entire desktop (e.g. for a transition time period, until apps are certified to run on the new platform.)
6. Corporate mergers and acquisitions – Client virtualization can be used to run two operating environments. In one VM, the image and application set of the acquirer while in another VM, the image and application set of the acquired company.
5. Trusted browser – Have you ever wanted to just isolate your browser so that anything you download can’t infect the rest of your machine? With client-hosted virtualization, you can and with Type 1, you can do it in a 100% secure environment. Also, with trusted browsing, you can be assured that if you are doing your online banking or other sensitive application, you can be assured there is no malware or phishing infecting it.
4. Virtual Software Appliances - the ability to package an OS, hardware dependencies, applications, user state, and security credentials into a single VM that can be deployed via multiple ways such as through traditional software delivery solutions, USB keys, DVD, or other method. For instance, a healthcare provider can package up a doctor’s computer that contains several applications such as X-ray records, patient information, diagnostic tools, and other sensitive apps and data into a single, encrypted VM. This can be packaged on a USB key that contains a fingerprint reader that allows the doctor to plug this key into any computer, swipe her finger to validate her identity and work on the applications, just as though she is in her office at the hospital.

3. Multi-project Desktop Consolidation – Government agencies and corporate users often provide their end users with multiple computers running specialized environments or applications that must (either due to technical limitations or compliance reasons) run on their own computer. This is expensive, not only for the hardware costs, but also the management costs associated with managing the extra machines. With Neocleus, you can run the each project in its own protected space, effectively working as a separate piece of hardware.
2. Remote / Disconnected VDI – Many companies are implementing Virtual Desktop Infrastructure using data center virtualization solutions to run hosted copies of users’ desktops in the cloud (internal or external), creating a nice secure platform for many types of users and applications. Using Neocleus, VDI customers can “check out” VDI images to take them on the road with them on their laptops.
1. Reduce the cost of laptops and desktops by allowing fast remediation of Windows and other OSes when they fail. In the event that Windows becomes compromised (e.g. the dreaded Blue Screen Of Death), you can have a VM running outside of Windows that can connect to the corporate network, download OS images, restart Windows, and get the user / machine up and running again immediately.

By the way, if you’re ever in NY, go ahead and stop by the Letterman show. It’s free and a lot of fun. If you smile when you pick up your ticket, they might even put you in the front row… (so I’ve heard ;-) )

 - Bill Corrigan

Leave a Comment » | Business Solutions, Solutions, Technology, Vision | Permalink
Posted by billcorrigan

Getting serious about a Client Hypervisor

January 22, 2009

We are pleased with Citrix and Intel announcing their intentions to further advance XCI. As one of the founding members of XCI, we believe it and similar open source initiatives can be a leading force in the industry creating an eco-system of third-party ISVs around type 1 client virtualization delivering applications and supporting client usage models which can only be delivered with virtualization.

The keys to making this successful are:

  1. Leverage the collective development efforts of the community
  2. Be a catalyst for innovation: Xen needs to be the core of a ubiquitous, possibly commoditized, client hypervisor allowing others to deliver best-of-breed client virtualization solutions

Neocleus has presented device pass-through as the cornerstone of delivering a best-of-breed type 1, bare metal, client hypervisor while others have failed to recognize it. Citrix and Intel’s announcement today publicly reaffirms this. Additionally, we are pleased to see that Citrix has validated one of our most prevalent targeted use cases of Bring-Your-Own-Computer (“BYOPC”) as they believe this will also be a major way that customers will derive benefits from the client hypervisor platform.

We expect to fully cooperate with Citrix, Intel and others in the industry, including VMware and Microsoft, in making the concept of client hypervisors a reality that provides exhaustive benefits to our customers. We also see this announcement as another possible forcing function for the industry to look at how software and OS licensing should change to accommodate this new world where a user can have multiple instance of OSes and applications running on their one computer.

Yes, in some areas we will be competing with Citrix yet we believe that our undivided focus on client virtualization solutions in the areas of security and a flexible management will allow us to build an independent and successful company. Neocleus is well positioned to be a leader in this market, as we continue to innovate and overcome the technical obstacles in delivering a full pass-through client hypervisor plus device switching capabilities. This is necessary to provide the native OS look and feel that will make the client hypervisor a viable platform for end users and IT.

In the meantime, our customers are excited by our upcoming of Neocleus’ 2.0 client virtualization solution, currently available in Preview stage to select customers and Partners, presenting side-by-side Windows plus our central management system.

Leave a Comment » | Contenders, Eco-System, Intel, Microsoft, VMWare, Xen | Permalink
Posted by Etay Bogner

Design Principles of a Client Hypervisor

January 20, 2009

When considering the design principles of client hypervisors and client virtualization solutions, one needs to look at the variety of available solutions today and understand how to make the quantum leap in order to bring a better product to the market (since it wasn’t available before).

When considering all use cases, most of our customers are already aware of Client-Hosted, Type 2, Client Virtualization solutions. Those solutions suffer from architectural limitations which prevent them from being used as a ubiquitous client hypervisor.

Type 2 Client Virtualization solutions are typically installed on top of a user-accessible operating system such as Windows. The flaws of this architecture are in the areas of security (if the “host” is infected, so is the “guest”), performance (since the “Guest” is competing for hardware resources as any other normal process) and hardware compatibility (the “guest” is using a completely emulated or para-virtualized device model).

The common knowledge advantage of Type 1 is its security architecture. Other not-so-easy to realize advantages are performance and hardware compatibility. Performance is realized because a Type 1 client hypervisor can be better tuned than a Type 2 client hypervisor providing more granular resource allocation to a specific virtual machine.

A Type 2 client-hosted hypervisor has one architectural advantage over Type 1 client hypervisors: It is non-intrusive. If you do not use any virtual machines, no runtime overhead is incurred on the “host”. The primary disadvantage associated with Type 1 is therefore that it is intrusive, in that it must be permanently installed on the bare-metal hardware, below the OS.

From an architectural point of view, Type 2 client-hosted hypervisors shortcomings cannot be resolved, and, a Type 1 Client Hypervisor is intrusive by design. The question is then, how to design a best-of-breed client hypervisor? It is evident that architecturally, a Type 1 client hypervisor has the most promise. Therefore, the critical success factor would be to overcome the intrusiveness and I would also add “transparency” to the mix. Other properties such as Security and Performance are intrinsic, in terms of design, to Type 1 hypervisors.

The intrusiveness of Type 1 hypervisors in general and client hypervisors in particular stems from the fact that the current design of Type 1 hypervisors is to in effect control most if not all of the hardware platform and devices and provide a near complete emulated and/or para-virtualized device model to the virtual machines running on top.

The Type 1 virtualization solutions available today make perfect sense for server virtualization but not for client virtualization. Server virtualization is all about virtualizing I/O, mainly disk and networking, but not CPU and human “accessible” devices. Simply put, client virtualization must take into account that there is an end-user using the PC. Consequently, the specific criteria which a client hypervisor would be measured against would be native device support which directly affects the user experience and expectations.

Hence, Type 1 client hypervisors must support full device pass-through to the “guest” virtual machines. Not only that, a best-of-breed Type 1 client hypervisor will also support dynamic assignment and “switching” of devices between different “guest”s. In effect, without device pass-through, Type 1 client hypervisor solutions render the virtual machines as “pale” as Type 2 client hypervisors since both only provide an emulated and/or para-virtualized device model – and remember, this is a permanent installation!

By providing full device pass-through capabilities, a best-of-breed Type 1 client hypervisor will allow end-users and organizations the freedom to mix and match different device models in order to best fit their expectations and requirements.

Yes, some use cases require having a hardware agnostic virtual machine that can be deployed universally but when considering all use cases, a Type 1 client hypervisor must feature a “modus operandi” that will reflect the real capabilities of the device since otherwise our customers tells us that it doesn’t meet their requirements.

Leave a Comment » | Architecture, Research | Permalink
Posted by Etay Bogner

Missing the Point of Client Hypervisors

January 16, 2009

What is important in providing a best-of-breed Client Hypervisor?

Virtualization is not a new concept. Specifically, device pass-through, device para-virtualization and device emulation are all well known concepts and viable technologies that are used in virtualizing operating systems; each has pros and cons from the various aspects of technology, performance, compatibility and maintenance.

Neocleus’ Client Hypervisor architecture is based on the understanding that a best-of-breed Client Hypervisor needs to address and support multiple use cases hence must utilize and apply different device virtualization technologies where and when they fit. Being “religious” about a specific technology will not help in addressing all use cases that our customers need to solve.

Technology wise, there is no contradiction between device pass-through, device para-virtualization and device emulation; all of them can and probably will be utilized together. Our prediction is that, over time, device emulation will fade out as para-virtualization delivers better performance.

Neocleus delivers what our customers are asking for: a flexible Client Hypervisor which in turn is managed by a central management system to address each and every Enterprise use case — all in order to optimize the end-user experience.

Let’s see how a best-of-breed Client Hypervisor addresses two prevalent use-cases:

  • Bring-your-own-PC: This case is about employees using their desktops at home or personal laptops to either work a couple of hours at home or full-time. This scenario presents the problem of how to maintain the performance and user experience of the “personal” environment (Windows or Mac OS in most cases – your kids will want to continue to play their high-end graphics games) while allowing the corporation to provision a corporate image (which may or may not be hardware agnostic). With Neocleus’ best-of-breed Client Hypervisor, the personal operating system can fully utilize the physical devices, taking advantage of our unique Device Pass-Through technology. From an IT perspective, each organization can decide how to deploy the corporate image (which is the second one running on the device) meaning deciding what is more important: User Experience (using physical devices if supported by the hardware) or being hardware agnostic – Neocleus’ client virtualization solution supports both (and even hybrid modes).
  • Corporate Laptops and Desktops: Here, again, a best-of-breed Client Hypervisor and the accompanied management solution will allow an organization to better serve its needs in deploying their corporate images and optimize them even on a per end-user request or role. For example, end-users and/or devices can utilize a full Device Pass-Through operating system because their work flow requires various USB devices and peripherals such as webcams for video conferencing; which may be accompanied by a second virtual machine for non-corporate Internet access which may be para-virtualized.

In short, a best-of-breed Client Hypervisor must utilizes and apply various technologies such as device pass-through and device para-virtualization in order to deliver the best user experience possible – custom-made to the different needs and requirements of IT and end-users. Additionally, with Neocleus Client Virtualization solutions organizations can enjoy the full flexibility of addressing multiple end-user scenarios all controlled from a centralized, scalable management system.

Leave a Comment » | Architecture, Vision | Permalink
Posted by Etay Bogner

« Previous Entries