Nmap Development mailing list archives
Re: Password Profiling and Password Mangling Libraries
From: Wong Wai Tuck <wongwaituck () gmail com>
Date: Mon, 03 Jul 2017 19:22:08 +0000
Hi George Dan proposed to include all profiling code to unpwdb (instead of a separate pwdprofile library) [1] and I too believe it's better that way I wanted to separate the libraries to separate the responsibilities of the libraries - one took care of loading username/password from files (and keeping them as two separate tables) and provides the interface for iterating through them, and the other one (pwdprofile) just solely looks at storing interesting candidates (regardless of whether they are usernames or passwords) and keeps its own list. However, I am fine with integrating it into unpwdb, since unpwdb isn't really that big! I'm not sure I understand the "pwdprofile.parse_html" method. Parsing or decoding various formats is not part of the password profiler. It is the script's responsibility to parse or decode this data and then pass any interesting candidate phrases to profiler (e.g. with your "add_phrase" method). I was thinking from the perspective of httpspider, and how it might be similar to CeWL. Many scripts use httpspider and I felt it didn't make sense to implement it in httpspider.lua (since logic of password profiling would then be in httpspider), and rather it would be best implemented in pwdprofile (and we additionally provide a flag to activate it), and letting httpspider simply pass the response to pwdprofile for processing. Do let me know your thoughts on this! Wai Tuck On Mon, Jul 3, 2017 at 10:25 PM George Chatzisofroniou <sophron () latthi com> wrote:
Dan proposed to include all profiling code to unpwdb (instead of a separate pwdprofile library) [1] and I too believe it's better that way. I'm not sure I understand the "pwdprofile.parse_html" method. Parsing or decoding various formats is not part of the password profiler. It is the script's responsibility to parse or decode this data and then pass any interesting candidate phrases to profiler (e.g. with your "add_phrase" method). [1]: http://seclists.org/nmap-dev/2017/q1/48 George On Sun, Jul 2, 2017 at 12:22 AM, Wong Wai Tuck <wongwaituck () gmail com> wrote:Hi list, I will be working on the password profiling library and password mangling library in the coming weeks. This is a follow up from the improvementsanddiscussions first proposed by my mentor George [1] and the improvements suggested by Dan [2]. Below shows the design plan for the two libraries. pwdprofile.lua The aim of this library is to keep track of possible username/password candidates that are found from other scripts (e.g. http-title,http-passwd)so that they can be used in the later part (for bruteforcing or to export out to another program). pwdprofile.lua will also help to keep track of which are the information gathering scripts that should run first, sothatthey can simply be simply included into current brute scripts (e.g. ftp-brute). By default, all information gathering scripts will log the candidates to the library, and an optional argument will allow the usertospecify if he wishes export the list out to a csv file. Script Args pwdprofile.export_file: the file to export data to, Default: nil pwdprofile.disable: disables pwdprofile, no new candidates will be added, Default false pwdprofile.custom_file: the file to import candidates from, Default nil pwdprofile.by_subnets: comma separated list of subnets that passwordswillbe aggregated by, rather than on a global basis Default nil pwdprofile.by_host: enables aggregation by host, rather than a global aggregation, Default false Script Constants: pwdprofile.STOP_WORDS_EN: a table of stopwords common in the English language pwdprofile.PWDPROFILE_SCRIPTS: a table of scripts that adds password candidates to pwdprofile Script methods: pwdprofile.add_word(host, keyword): saves a word verbatim to the password profile table. As new words are added, if the export option is specified, the word is also added to the file. pwdprofile.add_phrase(host, phrase, include_stop_words): parses a phraseforunique words, optionally including stop words like 'the' [3]. Callsadd_wordto add words. Default value for include_stop_words: false pwdprofile.parse_html(host, html, include_stop_words): parses the givenhtmlpage (as a string) for unique words, optionally including stop words like 'the'. pwdprofile.get_candidates(host): returns the table containing the username/password candidates that apply to the current host, given the arguments, whether it is global (default), by subnet, or by host pwdmangle.lua The aim of this library is to provide an interface to stream possible username password candidates into the iterator in unpwdb. By default, no mangling is done so it streams whatever was originally in unpwdb. Whenrulesare provided, it will mangle the passwords provided by unpwdb and pwdprofile, applying the rules given to generate additional words on theflythrough lazy evaluation. While I have yet to write something like this before, I believe it can be done through a stateful class, where eachbrutescript will call unpwdb to create a new instance of pwdmangle, and stream the password accordingly, so ideally the code for the brute scripts won't change, but changes will be made to unpwdb such that the next suggested password to use will be from pwdmangle. Script Arguments pwdmangle.rules: file that specifies the rules to apply. Rules canincludeour own (e.g. 1337 speak substitution) and those used by passwordcrackers[4]. In the new domain specific language we specify the rules in, weshouldbe able to combine lists, specify which particular list we are mangling(theunpwdb provided one, the pwdprofile, or the custom wordlists via theirindexin the comma separated arguments) or if we are mangling all listsglobally.Default nil pwdmangle.wordlists: custom wordlist(s) provided by the user,comma separated; Default nil pwdmangle.ignore_default: ignores the default list given by unpwdb.Default:false pwdmangle.export: exports the passwordlist generated out to the file provided. Useful for debugging passwords Class Mangler(host, unpwdb_passwds, wordlists, rules): constructor that creates a new instance of Mangler based on the host, the password listsandthe rules specified in the arguments. The default instance simply doeswhatunpwdb does currently, and doesn't do any mangling. Otherwise, itcreates astateful instance which keeps track of what the next password should be based on what has been generated before and the rules given, anddynamicallygenerate them. mangler.get_next_password() returns the next password candidate based onthecurrent state of the object, or nil if there are no more passwords to iterate through. The original passwords (given by unpwdb and pwdprofileareprioritized first, followed by the rules applied). unpwdb Modifications will be made to unpwdb to get passwords from pwdmangle, passing in the original table so that if the user wants to apply the mangling strategies to the original password list they can do so as well, without breaking support for existing scripts users may have written. The dependencies will be as such brute-script ----depends on----> unpwdb ---> pwdmangle --->pwdprofile Scripts affected by this change discover/version/vuln/exploit scripts (adding useful information to pwdprofile) *-brute scripts (adding dependencies) unpwdb Do let me know if you have any feedback for the design plan for the two libraries! [1]: http://seclists.org/nmap-dev/2016/q2/46 [2]: http://seclists.org/nmap-dev/2017/q1/67 [3]: https://en.wikipedia.org/wiki/Stop_words [4]: https://hashcat.net/wiki/doku.php?id=rule_based_attack With Regards Wai Tuck _______________________________________________ Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
_______________________________________________ Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Password Profiling and Password Mangling Libraries Wong Wai Tuck (Jul 01)
- Re: Password Profiling and Password Mangling Libraries George Chatzisofroniou (Jul 03)
- Re: Password Profiling and Password Mangling Libraries Wong Wai Tuck (Jul 03)
- Re: Password Profiling and Password Mangling Libraries George Chatzisofroniou (Jul 05)
- Re: Password Profiling and Password Mangling Libraries Wong Wai Tuck (Jul 03)
- Re: Password Profiling and Password Mangling Libraries George Chatzisofroniou (Jul 03)