Detection of repetitive and irregular hypercall attacks from guest virtual machines to Xen hypervisor

Virtualization is critical to the infrastructure of cloud computing environment and other online services. Hypercall interface is provided by hypervisor to offer privileged requests by the guest domains. Attackers may use this interface to send malicious hypercalls. In the reported work, repetitive hypercall attacks and sending hypercalls within irregular sequences to Xen hypervisor were analyzed, and finally, an intrusion detection system (IDS) is proposed to detect these attacks. The proposed system is placed in the host domain (Dom0). Monitoring hypercalls traffic the system operates based on the identification of irregular behaviors in hypercalls sent from guest domains to hypervisor. Later on, the association rule algorithm is applied on the collected data within a fixed time window, and a set of thresholds for maximum number of all types of the hypercalls is extracted. The results from the implementation of the proposed system show 91% true positive rate.