Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning
Current state-of-the-art research for tackling the problem of malware detection and classification is centered on the design, implementation and deployment of systems powered by machine learning because of its ability to generalize to never-before-seen malware families and polymorphic mutations. How...
| Authors: | ; ; ; |
|---|---|
| Format: | Article |
| Status: | Submitted version |
| Publication date: | 2022 |
| Country: | España |
| Institution: | Universitat de Lleida (UdL) |
| Repository: | Repositori Obert UdL |
| OAI Identifier: | oai:repositori.udl.cat:10459.1/72778 |
| Online Access: | https://doi.org/10.1016/j.cose.2021.102543 http://hdl.handle.net/10459.1/72778 |
| Access Level: | Open access |
| Keyword: | Malware Classification Assembly Language Source Code Obfuscation Reinforcement Learning Deep Q-Network |
| _version_ | 1878735338884562944 |
|---|---|
| author | Gibert Llauradó, Daniel |
| author2 | Fredrikson, Matt Mateu Piñol, Carles Planes Cid, Jordi |
| author2_role | author author author |
| author_browse | Fredrikson, Matt Gibert Llauradó, Daniel Mateu Piñol, Carles Planes Cid, Jordi |
| author_facet | Gibert Llauradó, Daniel Fredrikson, Matt Mateu Piñol, Carles Planes Cid, Jordi |
| author_role | author |
| collection | Repositori Obert UdL |
| description | Current state-of-the-art research for tackling the problem of malware detection and classification is centered on the design, implementation and deployment of systems powered by machine learning because of its ability to generalize to never-before-seen malware families and polymorphic mutations. However, it has been shown that machine learning models, in partidular deep neural networks, lack robustness against crafted inputs (adversarial examples). In this work, we have investigated the vulnerability of a state-of-the-art shallow convolutional neural network malware classifier against the deat code insertion technique. We propose a general framework powered by a Double Q-network to induce misclassification over malware families. The framework trains an agent through a convolutional neural network to select the optimal positions in a code sequence to insert dead code instructions so that the machine learning classifier mislabels the resulting executable. The experiments show that the proposed method significantly drops the classification accuracy of the classifier to 56.53% while having an evasion rate of 100% for the samples belonging to Kelihos_ver3, Simda, and Kelihos_ver1 families. In addition, the average number of instructions needed to mislabel malware in comparison to a random agent decreased by 33%. |
| eu_rights_str_mv | openAccess |
| format | article |
| id | ES_7ac1ec36c33d02e6600aeee699653ac0 |
| institution | Universitat de Lleida (UdL) |
| instname_str | Universitat de Lleida (UdL) |
| network_acronym_str | ES |
| network_name_str | España |
| oai_identifier_str | oai:repositori.udl.cat:10459.1/72778 |
| publishDate | 2022 |
| publishDateSort | 2022 |
| publisher | Elsevier |
| publisherStr | Elsevier |
| reponame_str | Repositori Obert UdL |
| spelling | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning Gibert Llauradó, Daniel Fredrikson, Matt Mateu Piñol, Carles Planes Cid, Jordi Malware Classification Assembly Language Source Code Obfuscation Reinforcement Learning Deep Q-Network Current state-of-the-art research for tackling the problem of malware detection and classification is centered on the design, implementation and deployment of systems powered by machine learning because of its ability to generalize to never-before-seen malware families and polymorphic mutations. However, it has been shown that machine learning models, in partidular deep neural networks, lack robustness against crafted inputs (adversarial examples). In this work, we have investigated the vulnerability of a state-of-the-art shallow convolutional neural network malware classifier against the deat code insertion technique. We propose a general framework powered by a Double Q-network to induce misclassification over malware families. The framework trains an agent through a convolutional neural network to select the optimal positions in a code sequence to insert dead code instructions so that the machine learning classifier mislabels the resulting executable. The experiments show that the proposed method significantly drops the classification accuracy of the classifier to 56.53% while having an evasion rate of 100% for the samples belonging to Kelihos_ver3, Simda, and Kelihos_ver1 families. In addition, the average number of instructions needed to mislabel malware in comparison to a random agent decreased by 33%. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 847402. This research has been partially funded by the Spanish MICINN Projects TIN2015-71799-C2-2-P, ENE2015-64117-C5-1-R, PID2019-111544GB-C22, and supported by the University of Lleida. Elsevier https://doi.org/10.1016/j.cose.2021.102543 http://hdl.handle.net/10459.1/72778 |
| spellingShingle | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning Gibert Llauradó, Daniel Malware Classification Assembly Language Source Code Obfuscation Reinforcement Learning Deep Q-Network |
| status_str | submittedVersion |
| title | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning |
| title_full | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning |
| title_fullStr | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning |
| title_full_unstemmed | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning |
| title_short | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning |
| title_sort | Enhancing the insertion of NOP instructions to obfuscate malware via deep reinforcement learning |
| topic | Malware Classification Assembly Language Source Code Obfuscation Reinforcement Learning Deep Q-Network |
| topic_facet | Malware Classification Assembly Language Source Code Obfuscation Reinforcement Learning Deep Q-Network |
| url | https://doi.org/10.1016/j.cose.2021.102543 http://hdl.handle.net/10459.1/72778 |