A malware detector is a system that attempts to determine whether a program has malicious intent. In order to evade detection, malware writers frequently use obfuscation to morph malware. Malware detectors that use a pattern-matching approach (for example, commercial virus scanners) are susceptible to such obfuscations. In spite of the importance of malware detectors, there is a dearth of testing techniques for evaluating them. I will present the results of evaluating the resilience of malware detectors to various commonly used obfuscation transformations. I will also demonstrate that a malware writer can leverage a malware detector's weakness to extract the signature used by a detector for a specific malware.
The poor resilience to obfuscation indicates the need for new approaches to malware detection. The fundamental deficiency in the current pattern-matching approaches is that they are purely syntactic and ignore the semantics of instructions. I will present a technique for malware detection that takes into account high-level program behavior without an increase in false positives. This behavior-based algorithm incorporates instruction semantics to detect malicious program traits. Furthermore, the algorithm is resilient to common obfuscations, while maintaining a relatively low run-time overhead (a requirement for real-time protection). Experimental evaluation demonstrates that our behavior-based malware-detection algorithm can detect variants of malware due to their shared malicious behaviors.
Somesh Jha received his B.Tech from the Indian Institute of Technology, New Delhi in Electrical Engineering. He received his Ph.D. in Computer Science from Carnegie Mellon University in 1996. Currently, he is an Associate Professor in the Computer Sciences Department at the University of Wisconsin (Madison). His work focuses on analysis of security protocols, survivability analysis, intrusion detection, formal methods for security, and analyzing malicious code. Recently he has also worked on privacy-preserving protocols. He has published over 70 articles in highly-refereed conferences and prominent journals. He has won numerous best-paper awards. Somesh also received the NSF career award.