Post

A GPU-Accelerated Windows 11 IoT Enterprise Jump Host for Homelab RDP

Posted
By Liam Thomas-Chollier 12 min read

After passing the AMD Embedded Radeon E9173 through to a VM on one of my Dell Wyse 5070s, the obvious next step was actually using that GPU for something. I settled on turning the VM into a jump host: one Windows box I RDP into from wherever I am, which then has the GPU horsepower to render a real desktop instead of the usual laggy, blocky RDP experience — and gives me a single, controlled front door into the rest of the homelab instead of poking holes for every service.

Why Windows 11 IoT Enterprise specifically

The Wyse 5070 isn’t a powerful box, and this VM has to share it with whatever else is running on the same host. I gave the jump host 5GB of RAM, and that’s genuinely enough — because IoT Enterprise is a much leaner OS than a normal Windows 11 install:

  • No Microsoft Store, widgets, Copilot, or the general consumer shell bloat — it’s meant to run headless/embedded devices for years, not to look nice on a retail PC.
  • Noticeably lower baseline RAM and CPU usage at idle than a stock Windows 11 Pro/Enterprise install, which matters a lot when the host only has so much to give and the GPU is already spoken for by this one VM.
  • It’s still “real” Windows underneath — same driver model, same RDP stack — so nothing about running Windows apps or a full desktop is compromised. It’s just trimmed down.

What I actually use it for day to day is genuinely light: RDP-ing onward into another machine on the network, some light web browsing, and light VS Code work — literally including writing this post and pushing it to git from inside the jump host. On top of that, since it runs 24/7 anyway, it also handles some background homelab chores via Task Scheduler (like the AD profile picture sync from an earlier post) and drives a small dashboard for a desk-mounted thin client. None of that needs much horsepower, so a lean OS on a small allocation of RAM/CPU with the GPU doing the heavy lifting for rendering the session itself is a good fit.

The GPU side: same passthrough, new job

The passthrough setup itself is exactly what’s already documented in the Wyse/E9173 post — IOMMU enabled at the kernel level, the GPU and its audio function bound to vfio-pci, vendor-reset installed to work around the Polaris/Lexa reset bug, and hostpci0: 0000:01:00,pcie=1,x-vga=1 in the VM config. I’m not repeating all of that here; go read that post if you’re setting this up from scratch.

What’s different is what runs on top of the passthrough once the VM boots into Windows 11 IoT Enterprise instead of a Linux gaming VM: the AMD driver gets installed normally inside the guest, and then the interesting part is telling Windows to actually use that GPU for encoding the RDP session instead of falling back to software (CPU) encoding.

Turning on GPU-accelerated RDP

Getting the GPU actually used for the RDP session — instead of Windows falling back to software encoding — comes down to a handful of Group Policy settings under Remote Desktop Services. I followed an existing tutorial for this part rather than piecing it together myself, so I won’t pretend to remember every exact setting I toggled — but the standard, commonly recommended combination (and the one that matches what’s working here) is:

1
2
3
4
5
6

Computer Configuration
  > Administrative Templates
    > Windows Components
      > Remote Desktop Services
        > Remote Desktop Session Host
          > Remote Session Environment

With, at minimum:

  • Use hardware graphics adapters for all Remote Desktop Services sessions — Enabled, so Windows uses the passed-through GPU to render the session instead of the software “Microsoft Basic Render” adapter.
  • Configure H.264/AVC hardware encoding for Remote Desktop Connections — Enabled (“Always use”), so the GPU also handles encoding the video stream itself rather than the CPU doing it in software.

These are technically Session-Host policies, but IoT Enterprise ships the full policy set (unlike Pro), so they apply even without an actual RDS role installed.

Worth checking after the fact: don’t use dxdiag for this — while an RDP session is active, dxdiag on the guest will only ever show the Microsoft Remote Display Adapter (rdpidd.dll/wudfrd.sys), which is the Indirect Display Driver RDP installs to receive rendered frames. That’s expected and doesn’t tell you anything about whether the real GPU is doing the encoding.

The better check is Task Manager on the jump host itself, watched from the console (not through the RDP session you’re testing) — Performance tab → GPU 0 → Video Encode graph. It should sit at 0% when idle and spike while an active RDP session has real motion on screen (dragging a window, scrolling, video playback). Confirming the guest sees the actual card and not a virtual one is also worth a glance in AMD Radeon Settings (Overview tab) — it should list the real chipset (AMD Embedded Radeon E9173, in my case), not a generic display-only device.

Classic RemoteFX Graphics perfmon counters are mostly a dead end here — that’s leftover plumbing from the deprecated RemoteFX vGPU feature and doesn’t reliably reflect the modern AVC/H.264 hardware path on Windows 11.

Testing it for real: it’s more complicated than “GPU busy = working”

Dragging windows around does bump the Video Encode graph a couple of percent, which at least confirms the GPU is in the loop. But a genuinely more demanding test — a Firefox page with an animation — dropped the session to something like 2fps, which the encoder graph alone doesn’t explain.

I ran a first round of testing remotely (before I was back on the LAN), just using Task Manager during an active RDP session:

  • GPU 3D usage: ~30% during the animation — some load, but nowhere near maxed.
  • CPU: 40-45% — also not pinned.
  • RAM: ~90% of the 5GB allocated — high, and enough to suspect paging.
  • Disk: stayed at 0% active the whole time. The system disk is a virtual drive backed by a physical SSD on the Proxmox host, so this rules out disk-paging as the cause — if RAM pressure were forcing Windows to swap, the disk graph would have shown it, and it never moved.

So the jump host’s own resources — GPU, CPU, RAM, disk — are all accounted for and none of them are actually maxed out or causing an obvious bottleneck. That points the finger somewhere I hadn’t originally considered: the RDP pipeline or network path itself, rather than anything on the VM:

  • RDP’s own graphics handling may just be struggling with continuous motion. Even with H.264 hardware encoding, RDP is fundamentally tuned for UI-style redraws (static regions, occasional updates) rather than full-frame video-like animation. A codec/quality setting negotiating down under sustained motion, independent of available GPU/CPU headroom, is a real possibility.
  • Network bandwidth/latency between me and the jump host, since I was testing remotely — the encoded stream still has to travel over whatever path gets it to me (through NexTerm, over the internet) before it reaches my screen, and a constrained link there would throttle the experience regardless of how much local GPU/CPU is free.
  • NexTerm/Kasm’s own relay step is still the one I haven’t isolated yet. With GPU/CPU/RAM/disk all cleared on the host side, a software re-encode happening in the access layer becomes the leading suspect by elimination.

The next test has to happen from the LAN directly: bypassing NexTerm/Kasm with a native RDP client straight to the jump host’s IP, running the same animation, to see whether the problem follows (protocol/GPU-side issue) or disappears (confirming the relay layer as the culprit). I’ll update this post once that’s done.

The access side: NexTerm and Kasm in front, not RDP directly

The part I actually want to underline: the RDP port itself is never exposed to the internet. RDP has a long, well-earned reputation as one of the worst things to expose directly — it’s a constant scanning/brute-force target and has had its share of serious vulnerabilities over the years. So instead of a port-forward to 3389, I sit two things in front of it, and which one I use depends on what I’m actually doing:

  • Kasm Workspaces is my daily driver when I need the session to be fully functional. Its RDP client is noticeably more polished — copy/paste works reliably in both directions, multi-monitor is handled properly, and it supports printer redirection plus file upload/download with a progress bar. The trade-off is that Kasm itself is a heavier stack — it’s running containerized workspaces and its own backend, not just relaying a connection — so it uses noticeably more resources than NexTerm for effectively the same job of getting an RDP session into a browser.
  • NexTerm is the lightweight option. Just a plain RDP session in the browser, no printer redirection or file transfer, and copy/paste is inconsistent — it works sometimes in one direction (host → RDP or RDP → host) but not reliably both ways. What it lacks in polish it makes up for in being cheap to run, which matters given the whole point of this jump host is squeezing useful work out of a low-power thin client.

It’s also the one my girlfriend uses — she’s on macOS day to day, and every so often she needs a quick Windows environment for something that just doesn’t exist or doesn’t work properly on Mac: a Windows-only app, testing something in Office, or printing from a program that only ships a Windows driver. For that kind of short, occasional session, NexTerm’s bare-bones client is more than enough, and there’s no reason to spin up Kasm’s heavier stack just to open Notepad or Word for five minutes.

Both sit behind my own OIDC provider for authentication — same login across NexTerm and Kasm, no separate account sprawl — and the jump host’s RDP port is only reachable from inside that layer; nothing hits 3389 from the outside world directly. It’s the classic “don’t expose RDP, put something in front of it” advice, just implemented with self-hosted tools instead of a commercial VPN/bastion product.

Kasm itself is off most of the time to save power — no point running its heavier container stack around the clock for something used occasionally. I only start it back up when I actually expect to need it: for example, ahead of heading out for longer.

The jump host VM and NexTerm, on the other hand, both run 24/7 — this isn’t just a Windows box that sits idle waiting to be RDP’d into. It’s also doing actual work in the background via Task Scheduler: things like the AD profile picture sync script run on a schedule from it, and it drives a small dashboard that shows up on a PoE-powered tablet-style thin client sitting on my desk. So the VM being always-on is doing double duty — background homelab tasks plus the always- reachable dashboard — while Kasm’s extra weight only gets switched on for the occasional heavier RDP session on top of that.

How it fits together

  1. I connect to Kasm (or NexTerm, for lighter use) over HTTPS from wherever I am.
  2. That opens an RDP session, over the LAN/homelab network, to the Windows 11 IoT Enterprise VM.
  3. That VM has the passed-through AMD GPU doing H.264 hardware encoding for the session, so even a fairly detailed desktop feels responsive instead of the classic soft, laggy RDP look.
  4. From inside that jump host, I have a normal desktop with access to the rest of the homelab — no need to expose anything else directly either.

Gotchas

  • GPU-P vs. full passthrough matters here. Because this is full PCIe passthrough (not GPU partitioning), the GPU is entirely dedicated to this one VM while it’s running — there’s no sharing it with another guest at the same time. Fine for a single jump host; wouldn’t work if I wanted several GPU-accelerated VMs off one card.
  • The vendor-reset requirement doesn’t go away. Same Polaris/Lexa reset bug as the Linux VM — restarting the Windows guest without vendor-reset loaded on the Proxmox host leaves the GPU wedged until the whole hypervisor reboots.
  • Policy application isn’t always instant. The hardware-adapter and H.264 policies sometimes need an existing RDP session to be fully closed (not just disconnected) before they visibly take effect on the next connection — worth remembering before assuming the GPU setting didn’t work.
  • “GPU busy” isn’t the same as “GPU is the bottleneck.” A couple percent of Video Encode utilization during a window drag confirms the pipeline is wired up correctly, but it doesn’t rule out something downstream (an access layer like NexTerm/Kasm, RDP’s own handling of continuous motion, or network conditions) being the actual thing capping performance under real load — in my case, GPU/CPU/RAM/disk all came back clear during a stutter, which pointed away from the VM entirely.
  • Don’t skip the “why not just expose RDP” reasoning. It’s tempting to port-forward and call it done, especially on a homelab where nothing feels that high-stakes — but RDP’s exposure history is bad enough that I’d rather have one extra hop (NexTerm/Kasm) than one direct path from the internet to a Windows login prompt.

Net result: one Windows box with real GPU acceleration behaving like an actual local desktop over RDP, reachable only through an authenticated web front door instead of a raw RDP port sitting on the internet.

Homelab, Virtualization
windows-11-iot rdp gpu-passthrough proxmox nexterm kasm remote-access
This post is licensed under CC BY 4.0 by the author.