KAL's whitepaper explains how OS-Virtualization can help banks run Windows 10 on ATMs without a hardware upgrade

You can download a copy of the whitepaper in PDF format here.

Executive summary

In January 2020, support for Windows 7 will end and it must be replaced by Windows 10. This migration presents banks with a major problem: the expense and complexity of upgrading their ATM software and hardware.

The shift from Windows XP to Windows 7 in 2014 cost the global industry billions of dollars because upgrading the ATM operating system also meant upgrading the hardware. Banks are about to encounter the very same issue again when they must upgrade to Windows 10.

But there is an answer: OS-Virtualization. This uses a technology known as a hypervisor to separate the hardware motherboard from the operating system so that software drivers that are unsupported under Windows 10 can be supported by the hypervisor software instead.

Hypervisor technology removes the need to upgrade current hardware when ATMs move to Windows 10, protecting the investments made by 20,000 banks worldwide in software and hardware, while remaining compliant with PCI.

Not only is this technology important when migrating to Windows 10, it will become essential when managing future Long-Term Servicing Channel (LTSC) software releases under Windows 10 which could require even more frequent hardware upgrades.

Find out more in this white paper.

Introduction

ATMs have been around for more than 50 years but have changed slowly during most of that time.

For the first 20 years, ATM architectures were proprietary to the ATM manufacturer, with hardware and software coming from the same vendor. The first revolution took place around 1990 when IBM’s OS2 made inroads as an operating system for ATMs opening the ATM black box a little with the use of a standard operating system. Windows then took over as OS2 faded, becoming the standard operating environment with Windows NT first, followed by Windows 2000, Windows XP, Windows 7 and now Windows 10. Most of the world’s bank-grade ATMs run some form of Windows today.

A second revolution kicked off around the year 2000 with the introduction of the XFS standard. All the specialized hardware devices inside the ATM, such as cash dispensers and card readers, started to have standard software drivers built to the CEN XFS standard. This gave birth to the multivendor software era that separated software applications from the hardware.

We are now at the dawn of another revolution. It is OS-Virtualization and it solves a major problem for the industry.

Until now, each time the ATM operating system required a major version upgrade (eg from Windows XP to Windows 7 in 2014), a hardware upgrade was also needed. The expense to the global industry was huge. It cost billions of dollars to upgrade the world’s 3.5 million ATMs.

Once again, a new cycle of upgrades is due in 2020 when support for Windows 7 ends and it must be replaced by Windows 10. Unsurprisingly, banks are balking at spending billions more dollars on upgrading their ATMs yet again.

The problem with upgrades

2014 was a bad year for banks running ATMs. When Microsoft dropped support for XP, it soon became clear that banks had little choice but to upgrade all their ATMs to Windows 7.

For Microsoft, the ATM industry is small. Three million-plus ATMs globally is minor compared with the two billion or more PCs worldwide, many of which run Windows. PC users always have the option of either delaying upgrading their PCs or buying relatively low-cost motherboard upgrades – options which are not available to banks for their ATMs. ATM motherboard upgrades are expensive – as much as $4000 in small volume, plus perhaps another $1000 to send trained technicians to upgrade the ATMs at locations around the country.

As banks keep ATMs for 10 years or more, many of the older models cannot even be upgraded and need to be replaced. New ATMs can cost between $10,000 and $30,000 depending on their functional capability, plus the cost of physical replacement, which can be exorbitant for “through-the-wall” ATMs in a busy location such as central Paris, London or New York. An ATM operating system upgrade brings no direct customer benefit but represents a huge cost just for compliance.

ATMs are subject to regulatory requirements – especially PCI – that insist there can be no unsupported software in the chain of software components required to run an ATM. Many Asian banks ignored this risk. American and European banks quite rightly did not take that path and upgraded their network of ATMs. But many of them have avowed that the upgrade cycle must stop.

However, as the year 2020 approaches, banks find themselves once again in the same upgrade predicament. Microsoft is dropping support for Windows 7 and banks will have to upgrade their ATM networks to Windows 10.

In fact, things could get even worse after Windows 10. Microsoft has announced that W10 will be the “last” Windows. There will not be a Windows 11 or a Windows 12. Does that mean the upgrade dilemma disappears after Windows 10? There have been newspaper articles proclaiming the end to upgrades. But, unfortunately, the reality is the exact opposite.

The new strategy for Windows is to make OS upgrades and enhancements happen even more frequently than before. Major upgrades will arrive in the form of “LTSCs” – “Long-Term Servicing Channel” packages. Microsoft plans to release LTSCs every three years in the future. Could this mean that ATM hardware might need to be upgraded every three years? The short answer is probably yes!

How did we get here?

Many blame Microsoft for updating their software too often or not supporting the operating systems long enough, but consider where the global software industry is heading.

The concept of DevOps and advances in automated testing mean that software release cycles are getting quicker all the time. Indeed, daily releases are not unusual in some parts of the software industry. It is therefore clear that Microsoft cannot be stuck in a 7-yearly update cycle for Windows. In fact, the W10 LTSC channel recommended for ATMs will be Microsoft’s slowest cycle for Windows releases. The version of Windows for businesses in general is the SAC (semi-annual channel), which updates much faster (every six months) and Microsoft mandates those updates be deployed. SAC is therefore unlikely to be appropriate for ATMs.

These faster upgrade cycles cause massive stress on the Wintel ecosystem. Although Microsoft is committed to supporting old hardware with new OS updates, the opposite is not true. Hardware component vendors have little interest in updating their old software drivers to support new releases of Windows that arrive well after they finish driver development. (Microsoft is no different here – they too will not support new hardware with old OSs).

This is what causes the support problem with OS upgrades. Software drivers, such as Intel’s chipset drivers, support only the OS versions that are available at the time of release of the chipsets – not new OSs that might be released well after the chipset driver development has been completed.

Intel say they will support a maximum of two LTSCs with any chipset. Depending on the frequency of arrival of new LTSCs from Microsoft, this limits the lifetime of ATM hardware. Indeed, when Microsoft initially announced Windows 10 LTSCs, the plan was to release LTSCs every 12-18 months, while Intel said they would support only one LTSC per chipset. Policy seems to have changed since then, with Microsoft settling on a release cycle of three years per LTSC and Intel agreeing to support two LTSCs per chipset. LTSCs so far released have been “1507, 1607, 1809 and 19H1” – these names are in fact Microsoft’s release dates, ie July 2015, July 2016, Sep 2018 and H1 2019 – so LTSCs have arrived much quicker than the future promise of every three years.

As you can see, this situation causes a major headache for the ATM industry. The upgrade cycle in the future looks like it could be every six years, but it could easily be as little as every 12 months, depending on the way LTSCs, and support for them, evolve in the future. The root cause is that as software cycles get quicker none of the hardware vendors in the support chain wish to support new OS releases with their old hardware components and drivers – they have no intention to re-open the development of old software drivers to integrate with future OS releases.

Between a rock and a hard place

This leaves the ATM industry with few options. Banks are used to running their ATMs over a 10-year period and many banks run ATMs for even longer than that. In fact, ATM hardware maintenance has more in common with aircraft maintenance than PC maintenance. Aircraft can be used for several decades, but over that period many of the hardware components may have changed multiple times so that perhaps only the airframe is truly 30 years old.

Similarly, a 15-year-old ATM has probably had multiple changes of card reader, cash dispenser and perhaps the PC-core too over its lifetime. The tricky bit, however, is the operating system. If the OS is to be changed as regularly as Microsoft sends upgrades, the motherboard will also need to be upgraded along with the OS – at a significant cost to the bank. The alternative is to run an unsupported OS and risk being non-compliant with PCI, alongside very real security risks such as malware (and the associated bad press that accompanies security breaches).

Is there another answer? KAL has focused on this problem since 2014, evaluating the options, trying to understand the root causes, and assessing which doors might potentially lead to a long-term solution.

Finding a solution

The upgradeable ATM? No. 

An option we considered is whether the PC-core of an ATM could be made more “upgradeable”.

Imagine a scenario where a PC-core upgrade was as easy as changing a DVD in a DVD player? In fact, Intel supports this type of concept with their “compute card”. It’s a PC-core about the size of a credit card that can be easily swapped. However, this would require the wholesale redesign of ATMs worldwide and would still need a relatively costly hardware swap with onsite intervention.

Don't upgrade at all? No.

There are a significant number of ATMs that still run Windows XP today at some Asian banks. This option to simply not upgrade the OS and continue to run Windows XP on ATMs is clearly a risky strategy. Not only is it not PCI compliant to run unsupported software, it also exposes the bank and its customers to potential malware and cyber-attacks, as vulnerabilities in unsupported operating systems get exploited by criminal gangs. This should not be considered a serious option.

However, there is an alternative to “not upgrading” ATMs that can potentially work in the future with Windows 10 and subsequent releases of LTSCs.

Let’s assume that a bank does initially upgrade their ATMs to Windows 10 as required. The version of Windows 10 in 2019 is called Windows 10 LTSC 1809 and it works in conjunction with a chipset that supports that LTSC. Microsoft, Intel and the ATM industry will support that combination for 10 years. That might appear to solve the upgrade dilemma, but in fact it doesn’t. Consider:

• New Windows 10 LTSCs as they arrive cannot be run on that 2019 ATM – ie over the whole of the 10-year life span that the ATM would need to run the original LTSC.
• Now consider the annual ATM replacement cycle. As an example, a large bank with say 10,000 ATMs would replace 10% of its estate each year as the ATMs age. Each year the bank might purchase 1000 new ATMs that would get delivered with the latest LTSC from Microsoft and the current chipset from Intel. In 2023, for example, they would receive LTSC 23XX. While those new ATMs will run LTSC 23XX, the old ATMs converted in 2019 can only run LTSC 1809. After 10 years, the ATM network would potentially have 10 different LTSC plus chipset combinations running different OS versions and different feature capabilities.
• Although each ATM would have a supported software stack over a 10-year period, this would be achieved by not upgrading the OS as LTSCs arrive, resulting in a fragmented network with potentially many different OS versions across the network. This would be an untenable scenario for most banks.

Change Intel’s support strategy for ATMs? No.

Aravinda Korala, KAL CEO, and Mike Lee, CEO of ATMIA, jointly undertook a series of conversations with Intel to understand whether they would consider a special support regime for their chipset drivers for the ATM industry.

The conversation was led by Oania Wei and Alec Gefrides, General Manager, Transactional Retail Segment at Intel. We explored concepts such as special paid long-term support for Intel drivers for the ATM industry and source code licensing to the ATMIA of old drivers. In the end, it became clear that for Intel, ATMs are a small industry for which none of the options we proposed were viable. However, Intel said “Look at Linux again” and introduced KAL to Wind River, a Linux distribution company that was at the time also an Intel subsidiary.

Migrate ATMs to Linux? No.

Linux has been an option for ATMs for a very long time. There has been some success running ATMs on Linux in Brazil using ATMs manufactured in Brazil – but not elsewhere.

The global ATM industry is a relatively small market with just 3.5m ATMs worldwide. Supporting a fragmented market with both Linux and Windows was not commercially feasible for the vendors. Consider that globally there are approximately 20,000 banks. The decision of which OS to run on ATMs is fundamentally made by the bank – not by the vendors. Banks just will not allow an OS for which they do not have a policy to be connected to their internal network – period.

How much time will it take to convince 20,000 banks to accept Linux as the OS on their ATMs? You can guess the answer. Any business case for migration to a complete Linux software stack would need to contend with a very long period of market fragmentation. The upshot of that is that none of the major ATM manufacturers have been willing so far to invest in supporting both Linux and Windows on their ATMs through such a transition period. As the XFS drivers are controlled by the manufacturers, if these drivers are not ported to Linux it is then impossible for any software vendor to run a Linux application on an ATM, irrespective of how deep the software vendor’s R&D pockets may be.

Why consider Linux at all? Linux has one especially useful feature that breaks the support logjam. All Linux software drivers, including Intel’s chipset drivers, are open source under Linux. That means that companies such as Wind River and Red Hat from the Linux ecosystem have access to that source code and can provide support on a commercial basis. This solves the PCI problem for banks.

But it still leaves us with the dilemma of the cost to migrate all the world’s ATM software at thousands of banks from Windows to Linux.

The solution – A Eureka moment

Although Linux can solve the long-term software support problem, it has a huge disadvantage compared with Windows in that banks have invested billions of dollars into ATM software that runs on Windows. The migration cost would be enormous.

There is a technical hurdle in addition to the financial one. ATMs have hardware drivers that use the XFS standard that are implemented on Windows only. Even if a bank wished to migrate their software stack to Linux, it could not do that unless the hardware supplier was also willing to migrate their XFS drivers to Linux. Currently, no drivers exist for ATMs from the major vendors on Linux.

Is there then a way to combine the virtues of Linux and Windows?

KAL worked on this problem with Wind River from mid-2017 onwards. The KAL team was led by Aravinda Korala and Kit Patterson, while the Wind River team was led by Kevin Konkos and Davide Ricci. The answer we arrived at is to use a Linux Hypervisor to host Windows 10.

Linux supports a hypervisor technology called QEMU. QEMU runs in conjunction with KVM to exploit the virtualization hardware acceleration supported in Linux and is able to present an interface to Windows 10 to run as a guest operating system on Linux. The hardware drivers for the motherboard now come from the Linux kernel, but the application environment runs under Windows. Support for QEMU, KVM and the Linux drivers comes from the Linux community and companies such as Red Hat and Wind River. The Linux part can therefore be supported for any length of time provided the commercial conditions work for the Linux companies.

Windows can sit atop of Linux and be updated as often as Microsoft and the banks wish to update it – and all of that can happen remotely online, without an onsite visit to the ATMs. This solves the upgrade conundrum. It eliminates the need for enforced hardware upgrades caused by Windows upgrades or LTSCs, and banks can stay completely up to date and run the latest versions of all software. Freedom at last from the enforced hardware upgrade cycle.

How it works

What is a hypervisor and virtualization?

Virtualization is not a new concept. IBM first created virtual machines and virtualized environments on mainframes as far back as the 1960s. It is now widely used in almost every datacenter in the world.

Hypervisors allow multiple operating systems to run on the same hardware server. One of the best known is the hypervisor from VMware. Global datacenters run hypervisors from VMware, Red Hat etc to isolate the hardware from the operating environment.

For example, it would be possible to run Windows XP and Windows 7 simultaneously as guest operating systems on a VMware host-OS. They also allow servers to be remotely controlled and managed using the hypervisor. A running system can be frozen in its tracks, moved to a new hardware server, and restarted from where it left off, as if nothing happened in between.

Intel and AMD hardware virtualization

Software hypervisors rely on hardware magic from Intel and AMD to do their job.

In the early days, virtualization was solely a software technique. It relied on software emulation to allow a guest OS to run on a host OS. But the effort of emulating and translating hardware access commands from one OS to another was costly and slowed the systems appreciably.

Around 2005, Intel and AMD built hardware virtualization support into their new CPUs. These technologies, known as Intel VT-x and AMD-V, allow the guest OS executables to run natively on the CPU, but catch system calls so they can be processed by the correct OS. The upshot is that application software in the guest OS runs as if it had the whole CPU to itself without discernible impact on performance.

In KAL testing, performance impact was measured to be only around 2% for a virtualized ATM running both Windows and Linux when compared with running Windows natively on the same hardware without virtualization. That is indeed magic.

The diagram below shows what the new ATM software architecture looks like inside an ATM with OS virtualization technology:

The hypervisor runs on the bare metal PC-core, Windows 10 runs as a guest operating system on top of the hypervisor, and the application software along with the XFS SPs from the hardware vendor runs inside a Windows virtual machine. 

What does a virtualized ATM solution look like?

At a conceptual level, a virtualized software solution looks exactly like today’s software stack, but with the addition of a hypervisor between Windows and the hardware. But, of course, it is not quite that simple to implement a real solution. Let’s begin with the prerequisites.

Prerequisites for a virtualized ATM solution

Although hypervisors can run on ATMs that do not have built-in hardware support for virtualization by using software emulation, this is likely to be too slow for production use. Here, then, is a list of prerequisites for virtualization of ATMs:

1. The first requirement is “VT-x” on Intel motherboards and “AMD-V” on AMD motherboards. As CPUs with these hardware capabilities began to appear around 2006, any ATM older than that will not be able to support virtualization without a significant performance degradation. It is also likely that ATMs with these capabilities did not appear until well after 2006 as ATM hardware vendors often used older stocks of CPUs over a long period, so the cut-off is sometime after 2006.
   
   
2. The virtualization capability on CPUs that support it can be disabled by a BIOS setting. This would require an onsite visit to change this setting adding cost to implementing a hypervisor solution. Some ATM hardware vendors have BIOS management tools that can be used remotely. KAL would recommend this option to first test the BIOS configuration and then change that configuration remotely to enable virtualization. Newer ATMs are most likely to have this option already enabled by default.
   
   
3. Next, banks will need to select a hypervisor vendor. Three companies have already expressed their interest in supporting the ATM industry – Red Hat, VMware and Wind River. KAL has tested the hypervisors from all three companies and can confirm that they all work. Microsoft, too, has a hypervisor called Hyper-V which is a standard part of Windows and could theoretically work, but it suffers from the same drawback as Windows itself. As the driver source code is not available for third-party device drivers in Windows, unsupported drivers with Hyper-V have no way of being supported by Microsoft or someone else.
   
   
4. Finally, banks need to check that their ATM hardware and software vendors will support their software applications and drivers in a virtualized environment. For example, if their ATM software vendor is KAL, they need to check that KAL will support a virtualized environment – which we do – and they need to ensure that the ATM hardware vendor will support their XFS SPs in a virtualized environment. Banks need to ensure that virtualization support is built into all future bank RFPs. That is what happened in the year 2000 with XFS. Banks mandated* XFS support from all vendors and it changed the ATM world.

Ready to go live?

Well, not just yet. Fulfilling the prerequisites is not quite enough. Banks also need to complete an implementation project:

1. The new hypervisor solution needs to be tested to ensure that all the test scripts that were used to acceptance test the current solution continue to work when the OS is virtualized – it should.
2. They need to review their ATM security lockdown. The new environment changes the security envelope – they need to lockdown the hypervisor as well as the Windows environment.
3. They need to review the ATM monitoring system. Ideally, the hypervisor software should be monitored too, along with the rest of the system.
4. Banks will need to review the software distribution mechanism. Ideally, the changeover should happen completely with remote software distribution, without the need to send a technician to the ATM. The software distribution system needs to be able to deliver patches and updates to the hypervisor software, as well as to Windows and the Application – all ideally online (or via DVD if that is the only option available).

Once banks have completed these steps, they are ready to go live.

KAL is ready today.

Long-term support strategy

There is an extra software component in the ATM software stack for which banks now need to have a support contract. They need to ensure that hypervisor support is added to the list. Here is what they need from the hypervisor vendor:

• Commitment to support ATM hardware on a long-term basis – at least 10 years if not more.
• Support for motherboard device drivers under Linux so that as 3rd party software drivers under Windows for that hardware go out of maintenance with new LTSCs, they can be substituted using Linux open-source drivers via virtualization.
• Support for Intel and AMD chipsets for as far back as the hypervisor vendors can provide so that older ATMs on their network can be supported.
• Support for future LTSCs from Microsoft as they arrive, being mindful of potential software driver related problems for old hardware as pointed out above.

Ideally, the ATM Industry needs a single global hypervisor release from each of the vendors to support all ATM motherboard models worldwide.

Conclusion and way forward

Virtualization solves a dilemma for ATM deployers by breaking the link between Windows OS upgrades and PC-core hardware upgrades. It allows the OS and the hardware to be upgraded separately and thereby removes the huge disruption to ATM networks that will be caused when Windows 7 goes out of support in 2020.

The first bank in the world to test the virtualization concept with KAL, along with two hypervisor vendors, was a US bank. The ability to run the bank’s current software stack in a virtualized environment was demonstrated in a PoC (proof of concept) that lasted a few days.

The first European bank to test the concept with KAL was Česká spořitelna in the Czech Republic. Jiří Charousek appreciated the technology immediately and set up the first tests. Jiří said “We were concerned at having to upgrade our hardware so soon after the XP-W7 upgrade and we are really happy that virtualization provides us with an alternative option. ATM virtualization was a very natural idea for us as it builds upon a long-term Česká concept within the infrastructure area – virtualization is our strategy already and ATMs were the exception.”

This concept is a boon for hardware vendors too. They often purchase motherboards and chipsets in large quantities only to discover that a new OS upgrade makes their old hardware stock worthless. As virtualization breaks the tight link between motherboards and operating systems, KAL believes that this will save costs for ATM manufacturers too.

OS-Virtualization provides the way forward to eliminate the costly, expensive and time-consuming process of banks having to continuously upgrade ATM fleets to support a new operating system. It does not remove the need to upgrade hardware forever – but it does remove the need to upgrade ATMs just because of an OS update. Banks will still need to upgrade their PC-cores either because they are too old, or too slow, or because they wish to use future CPU capabilities to deliver exciting new services to their customers. Banks will agree that is a good reason to upgrade their hardware.

Here on, it is essential that banks mandate virtualization support in all their RFPs for ATM software and ATM hardware.

Our thanks

Many people contributed to this project and confirmation that it really works. KAL would like to thank them very much in helping us arrive at this solution.

• ATMIA: Mike Lee
• Citibank: Peter Kulik
• Česká: Jiří Charousek
• Intel: Oania Wei
• KAL: Kit Patterson, Andrea Vinci, Giuseppe Scardino
• Microsoft: Pat Telford
• Payment Redesign: Eric de Putter
• Red Hat: David Hutchison-Bird, Daniel Schaefer, Rich Feldman
• VMWare: Thomas Klouwer
• Wind River: Davide Ricci, Rick Anderson, Kevin Konkos

Follow up

We’d welcome your comments, questions and feedback. Join the blog discussion below.

You can download a copy of the whitepaper in PDF format.

If you have any questions regarding the whitepaper, please do not hesitate to contact us.