PRE-list | List of (automatic) protocol reverse engineering tools for network protocols | Reverse Engineering library

PRE-list is a Python library typically used in Utilities, Reverse Engineering applications. PRE-list has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. However PRE-list build file is not available. You can download it from GitHub.

| Name | Year | Approach used | |------|------|---------------| | PIP [[1]] #1) | 2004 | Keyword detection and Sequence alignment based on Needleman and Wunsch 1970 and Smith and Waterman 1981; this approach was applied and extended by many following papers | | GAPA [[2]] #2) | 2005 | Protocol analyzer and open language that uses the protocol analyzer specification Spec → it is meant to be integrated in monitoring and analyzing tools | | ScriptGen [[3]] #3) | 2005 | Grouping and clustering messages, find edges from clusters to clusters for being able to replay messages once a similar message arrives | | RolePlayer [[4]] #4) | 2006 | Byte-wise sequence alignment (find variable fields in messages) and clustering with FSM simplification | | Ma et al. [[5]] #5) | 2006 | Please review | | FFE/x86 [[6]] #6) | 2006 | Please review | | Replayer [[7]] #7) | 2006 | Please review | | Discoverer [[8]] #8) | 2007 | Tokenization of messages, recursive clustering to find formats, merge similar formats | | Polyglot [[9]] #9) | 2007 | Dynamic taint-analysis | | PEXT [[10]] #10) | 2007 | Message clustering for creating FSM graph and simplify FSM graph | | Rosetta [[11]] #11) | 2007 | Please review | | AutoFormat [[12]] #12) | 2008 | Dynamic taint-analysis | | Tupni [[13]] #13) | 2008 | Dynamic taint-analysis; look for loops to identify boundaries within messages | | Boosting [[14]] #14) | 2008 | Please review | | ConfigRE [[15]] #15) | 2008 | Please review | | ReFormat [[16]] #16) | 2009 | Dynamic taint-analysis, especially targeting encrypted protocols by looking for bitwise and arithmetic operations | | Prospex [[17]] #17) | 2009 | Dynamic taint-analysis with following message clustering, optionally provides fuzzing candidates for Peach fuzzer | | Xiao et al. [[18]] #18) | 2009 | Please review | | Trifilo et al. [[19]] #19) | 2009 | Measure byte-wise variances in aligned messages | | Antunes and Neves [[20]] #20) | 2009 | Please review | | Dispatcher [[21]] #21) | 2009 | Dynamic taint-analysis (successor of Polyglot using send instead of received messages) | | Fuzzgrind [[22]] #22) | 2009 | Please review | | REWARDS [[23]] #23) | 2010 | Please review | | MACE [[24]] #24) | 2010 | Please review | | Whalen et al. [[25]] #25) | 2010 | Please review | | AutoFuzz [[26]] #26) | 2010 | Please review | | ReverX [[27]] #27) | 2011 | Speech recognition (thus only for text-based protocols) to find carriage returns and spaces, afterwards looking for frequencies of keywords; multiple partial FSMs are merged and simplified to get PFSM | | Veritas [[28]] #28) | 2011 | Identifiying keywords, clustering and transition probability → probabilistic protocol state machine | | Biprominer [[29]] #29) | 2011 | Statistical analysis including three phases, learning phase, labeling phase and transition probability model building phase. See [this figure] img/biprominer.png). | | ASAP [[30]] #30) | 2011 | Please review | | Howard [[31]] #31) | 2011 | Please review | | ProDecoder [[32]] #32) | 2012 | Successor of Biprominer which also addresses text-based protocols; two-phases are used: first apply Biprominer, second use Needleman-Wunsch for alignment | | Zhang et al. [[33]] #33) | 2012 | Please review | | Netzob [[34]] #34) | 2012 | See [this figure] | | PRISMA [[35]] #35) | 2012 | Please review, follow-up paper/project to ASAP | | ARTISTE [[36]] #36) | 2012 | Please review | | Wang et al. [[37]] #37) | 2013 | Capturing of data, identifying frames and inferring the format by looking and frequency of frames and doing association analysis (using Apriori and FP-Growth). | | Laroche et al. [[38]] #38) | 2013 | Please review | | AutoReEngine [[39]] #39) | 2013 | Apriori Algorithm (based on Agrawal/Srikant 1994). Identify fields and keywords by considering the amount of occurrences. Message formats are considered as series of keywords. State machines are derived from labeled messages or frequent subsequences. See [this figure] img/autoreengine.png) for clarification. | | Dispatcher2 [[40]] #40) | 2013 | Please review | | ProVeX [[41]] #41) | 2013 | Identify Botnet traffic and try to infer the botnet type by using signatures | | Meng et al. [[42]] #42) | 2014 | Please review | | AFL [[43]] #43) | 2014 | Please review | | Proword [[44]] #44) | 2014 | Please review | | ProGraph [[45]] #45) | 2015 | Please review | | FieldHunter [[46]] #46) | 2015 | Please review | | RS Cluster [[47]] #47) | 2015 | Please review | | UPCSS [[48]] #48) | 2015 | Please review | | ARGOS [[49]] #49) | 2015 | Please review | | PULSAR [[50]] #50) | 2015 | Reverse engineer network protocols with the aim to fuzz them with thus knowledge | | Li et al. [[51]] #51) | 2015 | Please review | | Cai et al. [[52]] #52) | 2016 | Please review | | WASp [[53]] #53) | 2016 | Pcap files are provided with context information (i.e. known MAC address), then grouping and analysing (looking for CRC, N-gram, Entropy, Features, Ranges), afterwards report creation based on scoring. | | PRE-Bin [[54]] #54) | 2016 | Please review | | Xiao et al. [[55]] #55) | 2016 | Please review | | PowerShell [[56]] #56) | 2017 | Please review | | ProPrint [[57]] #57) | 2017 | Please review | | ProHacker [[58]] #58) | 2017 | Please review | | Esoul and Walkinshaw [[59]] #59) | 2017 | Please review | | PREUGI [[60]] #60) | 2017 | Please review | | NEMESYS [[61]] #61) | 2018 | Please review | | Goo et al. [[62]] #62) | 2019 | Apriori based: Finding „frequent contiguous common subsequences“ via new Contiguous Sequential Pattern (CSP) algorithm which is based on Generalized Sequential Pattern (GSP) and other Apriori algorithms. CSP is used three times hierarchically to extract different information/fields based on previous results. | | Universal Radio Hacker [[63]] #63) | 2019 | Physical layer based analysis of proprietary wireless protocols considering wireless specific properties like Received Signal Strength Indicator (RSSI) and using statistical methods | | Luo et al. [[64]] #64) | 2019 | From abstract: “[…] this study proposes a type-aware approach to message clustering guided by type information. The approach regards a message as a combination of n-grams, and it employs the Latent Dirichlet Allocation (LDA) model to characterize messages with types and n-grams via inferring the type distribution of each message.” | | Sun et al. [[65]] #65) | 2019 | Please review | | Yang et al. [[66]] #66) | 2020 | Using deep-learning (LSTM-FCN) for reversing binary protocols | | Sun et al. [[67]] #67) | 2020 | "To measure format similarity of unknown protocol messages in a proper granularity, we propose relative measurements, Token Format Distance (TFD) and Message Format Distance (MFD), based on core rules of Augmented Backus-Naur Form (ABND)." for clustering process Silhouette Coefficient and Dunn Index are used. density based cluster algorithm DBSCAN is used for clustering of messages | | Shim et al. [[68]] #68) | 2020 | Follow up on Goo et al. 2019 | | IPART [[69]] #69) | 2020 | Using extended voting expert algorithm to infer boundaries of fields, otherwise using three phase which are tokenizing, classifying and clustering. |.