OpenStack and NUMA placement

Nonuniform memory access (NUMA) is a memory architecture that provides different access times depending on which processor is being used. This architecture introduces the concept of local and remote for memory access based on which core is accessing it. That means local access is faster compared to remote access.

This is a useful feature for improving the performance of virtualized guests. Guests can be optimized to use specific NUMA nodes when provisioning resources. On most modern hardware, one can specify which NUMA nodes a guest can use for virtualization. As an example, by improving performance and reducing latency, the network functions virtualization (NFV) use cases can really take advantage of it.

As part of the new features for the OpenStack Juno release, NUMA placement is being implemented and the specification details can be found in the Launchpad blueprint “virt-driver-numa-placement.”

Currently, OpenStack lets guests freely use vCPUs and memory from any NUMA node. There is no fine-tuning to gain better performance for certain workloads.

In simpler terms, the OpenStack Nova “libvirt” driver will be enhanced to take advantage of the compute node’s NUMA topology and allow the cloud administrator to do the fine-tuning through Flavors extra-specs and image metadata. The compute scheduler will take that into consideration and select the host that has a topology most similar to the desired NUMA topology.

Workloads on IBM POWER8 machines running OpenStack and using PowerKVM as the hypervisor can be optimized using this NUMA placement feature. For more details about IBM PowerKVM please refer to the downloadable IBM Redbooks publication “IBM PowerKVM Configuration and Use.”

In OpenStack NUMA, tuning Flavors extra-specs will be set as follows:

nova flavor-key m1.large set hw:numa_policy=strict hw:numa_cpus.0=0,1,2,3 hw:numa_cpus.1=4,5,6,7 hw:numa_mem.0=1 hw:numa_mem.0=1

Similar to Flavors, images can be updated as follows:

glance image-update image_id –property hw_numa_policy=strict –property hw_numa_cpus.0=0,1,2,3 –property hw_numa_cpus.1=4,5,6,7 –property hw_numa_mem.0=1 –property hw_numa_mem.0=1

Note that for images we use an underscore (_) instead of a colon (:), as it is the naming convention for image properties.

Using the “virsh” command directly, it is possible to dump the domain XML and add the NUMA tune settings entry:

# virsh dumpxml domain_id > domain.xml

Then edit the XML, adding a new entry within <domain>:


<memory nodeset=’0’/>

The preceding change will select node 0 as the guest node. Memory will be entirely allocated in the node 0.

NUMA placement is an essential and powerful feature for tuning libvirt performance. It seems to be a good component available in OpenStack to enable some specific workloads on the cloud.

What do you think about this new feature? Please, feel free to comment or find me on Twitter @timello to share more great things that NUMA placement will allow in OpenStack.

Comments Off
Tiago Rodrigues de Mello

About Tiago Rodrigues de Mello

Tiago is a software development engineer based in Brazil. Currently, he is working on the IBM OpenStack initiative.
This entry was posted in OpenStack and tagged , , . Bookmark the permalink.