So I wanted to include a routine in the aim sniffer to dump the authentication challenge and response hash to allow for brute forcing of the users password. Took some time to find the correct documentation on the OSCAR protocol but once I did it was pretty straightforward. Below is the basic process.
1. Server sends user random number (pseudo of course), typically around 10 digits
2. User encrypts password using md5 algorithm
3. User appends digest of password to the challenge code and then appends the string “AOL Instant Messenger (SM)” and takes an md5 digest of this and sends it as the authentication value. So the authentication code basically looks like this
Server sends: 123456789
User sends: md5( 123456789 . md5( theUserPassword) . “AOL Instant Messenger (SM)”)
Since you’re already salting the password using the challenge from the server the AOL string seems a bit gratuitous but whatever. Below is an example perl script that you could easily throw into a loop and read in a dictionary file and compare that to the hash sent by the user. Keep in mind the default output from the print statement will be in ASCII and thus might be pretty ugly. I plan on putting the the routine in the aim sniffer soon and will then show an example cracking session.
use Digest::MD5 qw(md5 md5_hex md5_base64);
$pass = "theUserPassword";
$key = "2075130568";
$aim = "AOL Instant Messenger (SM)";
#chomp($pass);
$digest = md5($pass);
$newdigest = "$key$digest$aim";
print "Authentication digest is: ", md5($newdigest) , "\n";
Tyler Wrightson's Security Blog 