Re: DPDK and energy efficiency

[Brian Knight via NANOG <nanog () nanog org>]

On 2021-03-05 15:40, Eric Kuhnke wrote:

> For comparison purposes, I'm curious about the difference in wattage 
> results between:
>
> a) Your R640 at 420W running DPDK
>
> b) The same R640 hardware temporarily booted from a Ubuntu server live 
> USB, in which some common CPU stress and memory disk/IO benchmarks are 
> being run to intentionally load the system to 100% to characterize its 
> absolute maximum AC load wattage.

We've got a few more hosts waiting to be deployed that are configured 
almost identically.  I'll see what we can do.

I'm guessing those tests would pull slightly more power than the vEdge 
hosts, just because there's not much disk IO that happens on a 
networking VM.  These hosts have four SSDs for local storage.

> What's the delta between the 420W and absolute maximum load the server 
> is capable of pulling on the 208VAC side?
>
> https://manpages.ubuntu.com/manpages/artful/man1/stress-ng.1.html

Server PS maximum input wattage is 900W.  Present draw of 2.0A @ 208V is 
~420W, so 420/900 = 46.67%

> One possible factor is whether ESXI is configured to pass the pci-e 
> devices directly through to the guest VM, or if there is any 
> abstraction in between. For non-ESXI stuff, in the world of Xen or KVM 
> there's many different ways that a guest domU can access a dom0's 
> network devices, some of which can have impact on overall steady-state 
> wattage consumed by the system.

The 420W server has its interfaces routed through the ESXI kernel.  
We're moving quickly to SR-IOV on new servers.

> If the greatest possible efficiency is desired for a number of 1U 
> things, one thing to look at would be something similar to the open 
> compute platform single centralized AC to DC power units, and servers 
> that don't each have their own discrete 110-240VAC single or dual power 
> supplies. In terms of cubic meters of air moved per hour vs wattage, 
> the fans found in 1U servers are really quite inefficient. As a 
> randomly chosen example of 12VDC 40mm (1U server height) fan:
>
> https://www.shoppui.com/documents/9HV0412P3K001.pdf
>
> If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's 
> a possible load of up to 18.24W for just *one* 40mm height fan. And 
> your typical high speed dual socket 1U server may have up to eight or 
> ten of those, in the typical front to back wind tunnel configuration. 
> Normally fans won't be running at full speed, so each one won't be a 
> 18W load, but more like 10-12W per fan is totally normal. Plus two at 
> least two more fans in both hot swap power supplies. Under heavy load I 
> would not be surprised at all to say that 80W to 90W of your R640's 
> total 420W load is ventilation.

Which is of course dependent on the environmentals.  Fan speeds on our 
two servers are 25% for the 260W vs. 29% for 420W, so not much 
difference.  Inlet temp on both is ~17C.

I checked out another R640 heavily loaded with vEdge VMs, and it's 
pulling similar power, 415W, but the fan speed is at 45%, because inlet 
temp is 22C.

The TDP for the Xeon 6152 is 140W, which seems middle-of-the-road.  From 
the quick survey I did of Dell's configurator, the R640 can take CPUs up 
to 205W.  So we have headroom in terms of cooling.

> In a situation where you're running out of power before you run out of 
> rack space, look at some 1.5U and 2U high chassist that use 60mm height 
> fans, which are much more efficient in ratio of air moved per time 
> period vs watts.

Or ask the colo to turn the A/C lower ;)  (that moves the power problem 
elsewhere, I know)

Thanks,

-Brian