Re: password patterns

[Emin İslam Tatlı <eitatli () gmail com>]

Hi Anders,

thanks for the detailed comments. I would try to clarify some points.

I start from the last issue.


> > I see that section 3 contains 'ending with 1',
It is not about ending with "1", but ending with "1." (one appended
with zero). See the example "dark1.". The passwords ending with "1"
are ca. 9.3%.


> > how many Alpha strings were plain dictionary words?
As I already stated in the report, I did not check how many of the
32.6M passwords belong to dictionary words. But I checked manually
more than 10.000 passwords and the majority belonged to dictionary
words.

> > If a majority, then it seems possible to take a dictionary and create a set of rewrite rules.
yes, this is the idea behind the patterns. They can enable such future
attacks. For example, the pattern "alpha+digit" combination has 30%
usage ratio and can be a good target to try.


> >  Section 1.3 does not take patterns such as Alpha + Digit + Alpha into account
I stated at the beginning of the article that 60% of the passwords are
insecure (i.e. contain only alpha chars, only digit etc.). The patters
I analyzed are then relevant to the remaining 40%. With the analyzed
patters I could cover more than 35% of the remaining 40% passwords.
The remaining 5% might contain other patterns or no pattern at all. No
pattern means  unlogical patterns  which seem quite randomly
generated. Regarding new patterns, as you said, alpha+digit+alpha or
alpha+punct+alpha or digit+alpha+digit etc. can be some other
candidates. But it should be taken into consideration that there might
exist conflicts between the patterns. Consider the password
"il0veyou". This password is compatible with both replacement-pattern
(o->zero) and alpha+digit+alpha pattern as well.

Actually, in the "additional patterns" section, I tried to define more
patterns. I believe the analysis should be extended through this
direction and more and more sub-patterns should be revealed. This
would increase the success of the rewrite rules.

By the way, "alpha+digit" is different than "alpha+digit+alpha".
"alpha+digit" means at the beginning of the password there are one or
more alpha characters followed by one or more digits at the end.


> > What character set was used for passwords? ASCII (i.e. 7-bit), some 8-bit code or Unicode?
UTF-8.


> > And what about characters outside those in [:alpha:] [:digit:] [:punct:] ?  (Did the password alphabet include 
> > characters
> > such as carriage return or line feed or horizontal tab, for example?)
I checked and there are around 6 passwords which contain some
characters outside those in alpha, digit, punct. In addition, there
are ca. 500 passwords which are either empty or contain only a space
character. On the other hand, this might have happened due to some
errors in the delivered password text file.


> >  How did you select the patterns you examined?
It is firstly based on my experiences from real life projects,
colleagues, friends etc. Secondly, I analyzed ca. 10.000 passwords
manually and tried to define the main patterns (i.e. combination and
replacement) and additional patterns (keyboard sequences, dates,
etc.). And in the analysis, I did check how frequently these patterns
are used.


> >  An interesting comparison could also have been made by taking the rewriting rules of a password cracker such as 
> > > > John the Ripper, and compared them with your findings.

the work can be extended from this perspective, right.

Emin

On Mon, Sep 20, 2010 at 9:07 AM, Anders Thulin <anders.thulin () sentor se> wrote:
> On 2010-09-13 14:56, Emin İslam Tatlı wrote:
>
> > I have recently written an article about password patterns. What do
> > you think about the topic? Can it be the next generation attack
> > replacing dictionary attacks? Any comments are welcome.
>
>  Though the article reports some interesting observations, it does not
> go far enough to allow any significant results to be extracted. For one thing:
> how many Alpha strings were plain dictionary words?  If a majority, then it
> seems possible to take a dictionary and create a set of rewrite rules. But
> there is no report that it was.
>
>  There also seems to be at least one serious flaw: Section 1.3 does not take
> patterns such as Alpha + Digit + Alpha into account, but there is no
> explanation why that is.  This alone makes that part of the study suspect.
> Nor is it clear that the character sets you used for your study cover all
> the characters found in the passwords ... so there seems to be a possibility
> that a number of passwords may simply have not matched those patterns.
>
>  Here are some of the question I noted as I was reading it.
>
>  What character set was used for passwords? ASCII (i.e. 7-bit), some
> 8-bit code or Unicode? Or perhaps some even more  restricted? Without knowing what
> characters were permissible in passwords, it's difficult to evaluate if the statistics
> you report are generally applicable, or if they only apply to this particular
> system. Is space allowed, for example?  And what about characters outside those
> in [:alpha:] [:digit:] [:punct:] ?  (Did the password alphabet include characters
> such as carriage return or line feed or horizontal tab, for example?)
>
>  Another interesting question is if there were any password recommendations
> associated with this set. It often happens that such recommendations are
> followed very closely -- if it said, for example that users can make a password
> from a plain word, prefixed by '#1' or ending with '.,' , I would expect to find a
> large number of such passwords in the data set. And that, of course, is a bias in
> any statistics based on the set. Are there any such biases to take into account?
>
>  How did you select the patterns you examined?  Was it based on what you
> found in the file, or on your own assumptions or the tools that you used for the analysis?
> Was examination systematical (i.e. all combinations were examined), or ad hoc?  (Part of this
> question may be because it's not quite clear to a reader what you mean by, say, Alpha+Digit.  Is that
> <a single alphabetical character> followed by <a single digit> ? Or is it a <a sequence of
> alphabetical characters> followed by <a sequence of digits> ?  As you use the term 'character set',
> I initially assumed that you meant single characters, but I suspect that may not be what you
> intended.)
>
>  The password 'mekster11' fit both models.  Would 'foo12bar' also be an Alpha+Digit pattern? Or
> is that an Alpha + Digit + Alpha pattern? For some unexplained reason that particular pattern
> seems to be missing from section 1.3 -- it really needs to be explained why it was left out, I think.
> As it is, the impression is that you did not do a systematic examination of this type of pattern.
>
>  For this question, I'm assuming you mean 'sequence of'. So: I'm sorry you stopped there.  It also makes
> sense to examine the characteristics of each of the strings.  In this particular case, a useful
> question is: what digits *are* used when the string contains only one digit? Only two? Only three?
> (In the case of three digits, is '007' more common than other '00x' strings, for example?)
> I see that section 3 contains 'ending with 1', but it's not at all clear from the context if
> that pattern (covering only 3047 passwords) was applied to the entire set of passwords, or only to some subset
> of passwords, and in that case, what subset. Alternatively, what single digits do appear?
> The patern you mention (Ending with 1) 'dark1' seems to contain Alpha + Digit pattern to me. If that is right, the
> last digit must have very strange statistics, as there are almost 10 millions Alpha+Digit passwords, and so
> 'ending with 1' should produce at least one million 'ending with 1', assuming even distribution of the last digit.
> But you report 3047. That's odd ... you probably need to explain why.
>
>
>  An interesting comparison could also have been made by taking the rewriting rules of a password
> cracker such as John the Ripper, and compared them with your findings.
>
>
> --
> Anders Thulin      anders.thulin () sentor se      070-757 36 10 / Intl. +46 70 757 36 10

------------------------------------------------------------------------
This list is sponsored by: Information Assurance Certification Review Board

Prove to peers and potential employers without a doubt that you can actually do a proper penetration test. IACRB CPT 
and CEPT certs require a full practical examination in order to become certified.

http://www.iacertification.org
------------------------------------------------------------------------