New tool enables comprehensive evaluation of datacenter performance

Datacenters are used by many companies, institutions and other operations, including large-scale services such as e-commerce, search engines, online maps, social media, advertising and more. These datacenters co-locate workloads, which involves sharing datacenter resources to improve server utilization.

However, this can lead to performance degradation. In order to study this problem and find possible solutions, researchers need to have tools to evaluate workload composition. Although such tools have been developed in the past, they may only measure one aspect at the expense of other factors, limiting their usefulness.

A team of researchers from Tianjin University and Dalian University of Technology, both in China, have developed a benchmark suite for workload benchmarking, called SDCBench, to address previous issues and provide comprehensive analysis.

The research was published in Intelligent Computing on 7 September.

“Workload collocation can cause performance disruptions that can cause unpredictable performance degradation for cloud services, which not only reduces the user experience but also hurts resource efficiency in datacenters,” the author said. correspondent Laiping Zhao, associate professor at the Tianjin Key Lab of Advanced. Networking in the College of Information and Computing at Tianjin University, China.

To overcome this issue, researchers try to improve the isolation capability – which refers to the privacy concerns of resource sharing at datacenters – of cloud systems through both hardware and software approaches. However, the recommended solutions may require new software or hardware updates, which some cloud providers cannot or will not provide.

“The need for predictable service performance in datacenters brings new challenges and opportunities for cloud system design that seeks to improve resource utilization at the server level but does not hurt application level performance,” said Zhao.

“Unfortunately, the lack of a comprehensive set of workload collocation benchmarks makes studying this emerging problem challenging. A workload collocation benchmark can help cloud providers understand and leverage infrastructure isolation capabilities improvement, which increases their adoption by cloud users.”

The researchers developed SDCBench, a set of benchmarks for workload aggregation that includes 16 latency-critical – meaning that there must be a significant delay in response time – services and applications that span a wide range of cloud scenarios.

“SDCBench enables cloud tenants to understand the performance isolation potential in datacenters and choose their best-suited cloud services,” said Zhao. “For cloud providers, it also helps them improve the quality of service to increase their revenue.”

In conjunction with the introduction of the new series of benchmarks, the researchers propose the concept of latency entropy, inspired by the physics definition of entropy which means the amount of disorder within a system, to measure the uncertainty of cloud systems.

“When shared resource contention occurs between different applications, system behaviors become erratic and unpredictable,” Zhao said. “To help users understand the application performance changes with observable metrics, SDCBench defines the latency entropy which describes the tail latency variations to measure the isolation capability of the system.”

The researchers showed that SDCBench can simulate different cloud scenarios by colocating workloads with simple configurations. They also evaluated and compared latent entropy in major cloud computing providers using their benchmark tool.

According to Zhao, one of the most exciting aspects of the research is that the SDC Bench and a comprehensive framework based on it are publicly available.

“We have implemented a comprehensive evaluation framework based on SDCBench that can automatically configure, deploy and evaluate applications on cloud platforms, and that framework is open source and can be easily extended to new cloud systems, Zhao said.

Available at Smart Computing