Simple Things

How to query software program vendor websites for the program version using ruby

hexeract — Sun, 17 Jul 2011 10:37:59 +0000

This is a followup to my previous article.

The example in the previous post only checked three programs, but with more entries the overall execution time increases. With about 36 entries the execution time is around 20-40 seconds, depending on how fast the websites answer as each website is being accessed in sequence and waiting for each request to complete. This can and will add up. Additionally running the various unix commands in the pipe will not help towards overall efficiency.

So I decided to give it a try to make the script better and a hopefully a lot faster, especially when more websites are being accessed. My first thought was to try it with perl but parsing HTML with perl never quite worked for me when I wrote some test scripts. This may be due to me being inapt using HTML::TreeBuilder and/or XML::XPath. In any case, I was getting nowhere…

The second choice of a scripting language to use was ruby. As I already had done some website scraping using the ruby gems mechanize, nokogiri and hpricot I thought trying to convert the shell script into a ruby script would be easy. Little did I know :)

Despite the fact it only accesses some websites and stores the version number (and the program name) in a sqlite3 database I hope it can serve as an example on how to use threads, mechanize, nokogiri, sqlite3 and how to pass code to a function.

Adding more checks does not increase the overall runtime too much. A test with 36 entries results in an overall runtime between two to five seconds.

Finding the actual xpath is not that easy, firebug (Firefox) or the Developer Tools from Google Chrome can provide helpful hints but this is best tested in a small test script where also the version extraction ruby code can be written.

XPath and version extraction test script (example Metapad):

#!/usr/bin/ruby

require 'rubygems'
require 'mechanize'
require 'nokogiri'

# exit upon control-c
trap("INT") { exit 1 } 

# site data
url   = 'http://liquidninja.com/metapad/download.html'
name  = 'MetaPad'
xpath = '/html/body/table/tr/td/table/tr/td/table/tr/td/table/tr/td'

# and access the site 
agent                  = Mechanize.new
agent.history.max_size = 0 
agent.user_agent_alias = 'Mac Safari'
agent.read_timeout     = 3 
page                   = agent.get(url)
found                  = page.search(xpath)

# this is to make it easier to just "drop" in the source code
# into the "real script".
data = found

# display the "elements" we have hit to see if we are "close" with the xpath
    data.each do |node|
        puts "-- NODE --"
        puts node
    end
    puts "--- OUTPUT ---"

##### version extraction code 

versions = Array.new

data.each  do |node|
   versions.push(node.text) if node.to_s.include?('Version')
end

data = versions.first
data = data.strip.gsub!(/Version /, '');

#### version extraction code end
# assign the result back to 'result' 
result = data

# and print what we have
puts "#{name}: #{result}"
exit 0

The sqlite3 database table

sqlite> .schema win32
CREATE TABLE win32 (program varchar(30) PRIMARY KEY, cver TEXT, pver TEXT, err INT default '0');
sqlite>

Prior to writing to the database, an entry for the program must exist. This is an example on how to create an entry:

sqlite> insert into win32 (program) values ('MetaPad');
sqlite> select * from win32 where program='MetaPad';
MetaPad|||0
sqlite>

And here is the final ruby script that checks three vendor websites and writes to the sqlite3 database:

#!/usr/bin/ruby

require 'rubygems'
require 'mechanize'
require 'nokogiri'
require 'sqlite3'
require 'socket'

# exit upon control-c
trap("INT") { exit 1 }

# Setup the database
dbfile  = 'win32.db'
dbtable = 'win32'

# no database, no party
if not File.exists?(dbfile)
    puts "Error, could not find database: #{dbfile}"
    exit 10
end

# connect to the database
database = SQLite3::Database.new(dbfile)

# This hash will store the program name
# and its (scraped) version string
pv = Hash.new

# Using threads to make website scraping concurrent (and thus faster)
mythreads = Array.new

# This is the core routine of the script
# It takes three(!) arguments:
#  url:      What URL to access/get HTML from
#  xpath:    Using nokogiri, "extract" only a part of the HTML content
#  CODE:     The source code to parse the part of the HTML content to obtain
#            the version string

def getversion(url,xpath)
    # Set the default return value to ''
    result = ''

    # TCP connection check, host/service available?
    host   = url.split('/')
    host   = host[2]
    socket = TCPSocket.open(host, '80')

    # No socket, no access
    begin 
        socket = TCPSocket.open(host, '80')
    rescue  
        return result 
    end
    socket.close

    # This sets up a simple web client
    agent                  = Mechanize.new
    agent.history.max_size = 0
    agent.user_agent_alias = 'Mac Safari'
    agent.read_timeout     = 3
    agent.keep_alive       = false

    # Try and download the page
    # If something goes wrong, result will not have changed
    # and thus will be '' which will force the error counter to be increased
    begin
        page                   = agent.get url 
    rescue Mechanize::ResponseCodeError
        return result
    end

    # Parse the content using the xpath and nokogiri
    # which is accessed via the mechanize method .search 
    found                  = page.search xpath

    # If the array (of nokogiri) objects is not empty
    # execute the provided code block to obtain the version number
    if not found.empty?
    then
        result = yield found if block_given?
    end

    return result
end

###
# Metapad
t = Thread.new {
    url    = 'http://liquidninja.com/metapad/download.html'
    name   = 'MetaPad'
    xpath  = '/html/body/table/tr/td/table/tr/td/table/tr/td/table/tr/td'

# This is the call where url, xpath and some code is being supplied to the function
# The code passed uses the 'data' object which is the one also being 'returned'
# to the subroutine when the code block finishes

    result = getversion( url, xpath ) { |data|
                                        versions = Array.new
                                        data.each  do |node|
                                            versions.push(node.text) if node.to_s.include?('Version')
                                        end
                                        data = versions.first
                                        data = data.strip.gsub!(/Version /, '')
                                      }
    pv["#{name}"] = result
}
mythreads.push(t)

###
# Pidgin
t = Thread.new {
    name   = 'Pidgin'
    url    = 'http://developer.pidgin.im/wiki/ChangeLog'
    xpath  = '/html/body/div/div/div/div/div/h2'

# Simpler code to extract the version number
    result = getversion( url, xpath ) { |data|
                                        data = data.first.text
                                        data = data.split(' ')
                                        data = data[1]
                                      }
    pv["#{name}"] = result
}
mythreads.push(t)

###
# Adobe Flash Player
t = Thread.new {
    url    = 'http://www.adobe.com/de/software/flash/about/'
    name   = 'Adobe Flash Plugin'
    xpath  = '/html/body/div/div/div/div/table/tbody/tr/td'

# Simplest version :)
    result = getversion( url, xpath ) { |data| data = data[5].text }
    pv["#{name}"] = result
}
mythreads.push(t)

#### More entries could go here :)

# Collect all threads and wait, if necessary
mythreads.each { |t| t.join }

# Create a string of database commands by
# looping through the 'pv' hash and create the individual update commands for sqlite3
# Each command is one element of the 'sql' array
sql = Array.new

pv.each do |p,v|
    sql.push("update #{dbtable} set cver = '#{v}' where program = '#{p}';")
end

# Convert the 'sql' array to one big string and batch execute
database.execute_batch( sql.join("\n") )

# The cver column for each program has been updated
# What is left is to query the database and print the results, if there are any

# reset error counter when cver has a value
database.execute("update #{dbtable} set err = 0 WHERE cver != ''")

# Report if the err counter is >= 5
# Rhis means 5 consecutive runs went 'wrong' and requires a check as to # what/why.
# For example, if the vendor changes the output/structure of the website...
data = database.execute2("select program AS ProgramName,err AS ErrorRuns from win32 where err >= 5")

# When executing the method execute2 the table header is also returned
# this means, the data array will always contain one element
# It will contain a second (and more) element(s) if actual data is returned
if data[1]
then
    data.each do |line|
        printf("%-20s %-15s\n", line[0], line[1])
    end
end

# Increase the error counter if current version is empty
database.execute("update win32 set err = err  + 1 WHERE cver=''")

# Compare the current version with previous version
# Do not compare when current version is empty
data =  database.execute2("select program AS ProgramName,pver AS OldVersion,cver AS CurrentVersion from win32 where ( cver != pver and cver != '')")

# see remark about execute2 above
if data[1]
then
    data.each do |line|
        printf("%-20s %-15s %-15s\n", line[0], line[1], line[2])
    end
end

# Set the previous version to the value of current version
# Do not set if the current version is empty
database.execute("update win32 SET pver = cver WHERE ( pver != cver and cver != '')")

exit 0

How to keep up with new versions of programs

hexeract — Fri, 15 Jul 2011 15:40:38 +0000

Keeping up with versions on your computer can be quite tedious, especially if those programs do not provide an auto-update feature. You could visit the vendor homepage for each software product you intend to keep up to date and check manually or use a program and hope it knows how to check for updates for your software and last but not least visit a website regularly showing newly released/updated software.

Or you can build something yourself.

The example used here is awful and I am sure there a a million better ways to scrape websites (and as you will see later, can be easily replaced by something else) but should suffice to demonstrate on how to keep track of versions of programs you deem important.

Shown below is a method on how to scrape version information from vendor websites using curl and the usage of some unix commands to modify output from curl. If you want to know what each part in the pipe does just start with the initial command, look at the output and add a step at a time.

Example:

#!/bin/bash
echo -n "Metapad: "
curl -s -m 3 http://liquidninja.com/metapad/download.html | grep -A 5 "Latest Release"  | tail -1 | sed "s/Version //g"
echo -n "Pidgin: "
curl -s -m 3 http://developer.pidgin.im/wiki/ChangeLog | grep version | head -2 | tail -1 | perl -p -i -e "s/<.*?>//g,"  | awk '{ print $2 }'
echo -n "Adobe Flash Plugin: "
curl -s -m 3 http://www.adobe.com/de/software/flash/about/ | grep -A 2 "Firefox, Mozilla, Netscape, Opera"  | tail -1 | perl -p -i -e "s/<.*?>//g," | awk '{ print $1 }'

For the purpose of this article, the amount of checks is limited to three. Most likely you want to keep track of versions for more than three programs. Even if one assumes that you know the versions of all programs on your computer off by heart and you always install the latest software when available it is still easy to miss if an update is available just by looking at the script output. Especially with a lot more than just three…

In this scenario it would be helpful if the check could remember the version for each program from the previous run. Again, like with the example above on how to scrape websites, there are a million ways to keep track of such data, may it be a flat file (with either shell wizardry in-place editing/comparisons or using a revision control system) or a full blown Oracle Database 11g.

As I only want to store four values (program name, current version, previous version, error count) I decided to use sqlite3. This allows for a future migration to a proper database if the need arises.

Creating a database with sqlite3:

$ sqlite3 win32.db "create table win32 (program varchar(30) PRIMARY KEY, cver TEXT, pver TEXT, err INT default '0');"

Now, the database needs content. With just three examples, it would be faster to just manually enter the data.

Here are the individual commands:

$ sqlite3 win32.db "insert into win32 (program, cver, pver) values ('Metapad','3,6','3,6');"
$ sqlite3 win32.db "insert into win32 (program, cver, pver) values ('Pidgin','2.9','2.9');"
$ sqlite3 win32.db "insert into win32 (program, cver, pver) values ('Adobe Flash Plugin','10.3.181.34','10.3.181.34');"

Checking if all data is in the database:

$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad     3,6         3,6         0         
Pidgin      2.9         2.9         0         
Adobe Flas  10.3.181.3  10.3.181.3  0

As seen, the data is available. But as the check script requires the functionality to write to the sqlite3 database this can be exploited to fill the database with initial values.

At first, delete the database and recreate it from scratch:

$ rm win32.db
$ sqlite3 win32.db "create table win32 (program varchar(30) PRIMARY KEY, cver TEXT, pver TEXT, err INT default '0');"

For the script to write to the database, a function will do fine. The function takes the name and version as arguments, strips ^M and leading/trailing whitespaces from the version string and then writes the data to the database. The following shows the script that will fill the database with initial values.

Example:

#!/bin/bash

DATABASE="win32.db"

if [ ! -e ${DATABASE} ]
then
    echo "Error: Can not find sqlite3 Database to update/check"
    exit 10
fi
function updatedb()
{
    name=$1
    version=$2
    sver=$(echo $version | tr -d '\015' | sed 's/^[ \t]*//;s/[ \t]*$//')
    sqlite3 ${DATABASE} "insert into win32 (program, cver, pver) values ('$name','$sver','$sver');"
}

VERSION=$(curl -s -m 3 http://liquidninja.com/metapad/download.html | grep -A 5 "Latest Release"  | tail -1 | sed "s/Version //g")
updatedb "Metapad" "$VERSION"

VERSION=$(curl -s -m 3 http://developer.pidgin.im/wiki/ChangeLog | grep version | head -2 | tail -1 | perl -p -i -e "s/<.*?>//g,"  | awk '{ print $2 }' )
updatedb "Pidgin" "$VERSION"

VERSION=$(curl -s -m 3 http://www.adobe.com/de/software/flash/about/ | grep -A 2 "Firefox, Mozilla, Netscape, Opera"  | tail -1 | perl -p -i -e "s/<.*?>//g," | awk '{ print $1 }')
updatedb "Adobe Flash Plugin" "$VERSION"

Check if all the data has made it into the database as intended:

$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad     3,6         3,6         0         
Pidgin      2.9         2.9         0         
Adobe Flas  10.3.181.3  10.3.181.3  0

As you can see, the “Adobe Flash Player” string looks truncated.

Better check that:

$ sqlite3 -header -column win32.db "select * from win32 where program like 'Adobe%';"
program             cver         pver         err       
------------------  -----------  -----------  ----------
Adobe Flash Plugin  10.3.181.34  10.3.181.34  0

Only the output was truncated, not the actual value.

As the database now has an intial state, the script requires some modifications so it can use and update the database. The function updatedb only requires a minor modification but at the end of the script there are some sqlite3 commands that check the version (and failed checks) and outputs program names where there is an update available.

This is what the complete script looks like:

#!/bin/bash

DATABASE="win32.db"

if [ ! -e ${DATABASE} ]
then
    echo "Error: Can not find sqlite3 Database to update/check"
    exit 10
fi

function updatedb()
{
    name=$1
    version=$2
    sver=$(echo $version | tr -d '\015' | sed 's/^[ \t]*//;s/[ \t]*$//')
    sqlite3 ${DATABASE} "update win32 set cver='$sver' where program='$name';"
}

VERSION=$(curl -s -m 3 http://liquidninja.com/metapad/download.html | grep -A 5 "Latest Release"  | tail -1 | sed "s/Version //g")
updatedb "Metapad" "$VERSION"

VERSION=$(curl -s -m 3 http://developer.pidgin.im/wiki/ChangeLog | grep version | head -2 | tail -1 | perl -p -i -e "s/<.*?>//g,"  | awk '{ print $2 }' )
updatedb "Pidgin" "$VERSION"

VERSION=$(curl -s -m 3 http://www.adobe.com/de/software/flash/about/ | grep -A 2 "Firefox, Mozilla, Netscape, Opera"  | tail -1 | perl -p -i -e "s/<.*?>//g," | awk '{ print $1 }')
updatedb "Adobe Flash Plugin" "$VERSION"


sqlite3                 ${DATABASE} "update win32 set err = 0 WHERE cver != '';"
sqlite3 -header -column ${DATABASE} "select program AS ProgramName,err AS ErrorRuns from win32 where err >= 5;"
sqlite3                 ${DATABASE} "update win32 set err = err  + 1 WHERE cver='';"
sqlite3 -header -column ${DATABASE} "select program AS ProgramName,pver AS OldVersion,cver AS CurrentVersion from win32 where ( cver != pver and cver != '');"
sqlite3                 ${DATABASE} "update win32 SET pver = cver WHERE ( pver != cver and cver != '' ); "

The function updatedb was modified to update the current version column and five sqlite3 commands were added.

There is a fourth field in the database called ‘err’ which is an error counter. This is used to detect if the check for a version returns an empty string and for how many consecutive executions of the program.

Set the error counter to zero when the current version contains anything but an empty string.
Display the program name and error counter if the error counter is equal or greater than five.
Increase the error counter when the current version contains an empty string.
Show the program name, the previous version and current version if current version is not an empty string.
Set previous version to the value of current version when they are not equal and current version is not an empty string.

This should catch the problem when a check returns nothing. The check may be broken and needs attention/fixing.
In any other case the version string may be some odd text which can be easily spotted when the current version and the previous version differ.

To check if the script works as intended, the pver value for one program can be modified using the following command:

$ sqlite3 win32.db "update win32 set pver = 12345 where program like 'Adobe%';"
$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad     3.6         3.6         0         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  12345       0

Running the script now should provide an output:

$ ./checkscript.sh
ProgramName         OldVersion  CurrentVersion
------------------  ----------  --------------
Adobe Flash Plugin  12345       10.3.181.34

Also, the OldVersion should be overwritten by the value of CurrentVersion:

$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad     3.6         3.6         0         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0

A subsequent run of the script should not yield any output, unless the vendor updated the version in the time frame we ran the script again. But does the error counter work as intended? Let’s find out!

Manually setting the current version to an empty string:

$ sqlite3 win32.db "update win32 set cver='' where program = 'Metapad';"
$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad                 3.6         0         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0

Rerunning the five sqlite3 commands and checking the result:

$ sqlite3                 win32.db "update win32 set err = 0 WHERE cver != '';"
$ sqlite3 -header -column win32.db "select program AS ProgramName,err AS ErrorRuns from win32 where err >= 5;"
$ sqlite3                 win32.db "update win32 set err = err  + 1 WHERE cver='';"
$ sqlite3 -header -column win32.db "select program AS ProgramName,pver AS OldVersion,cver AS CurrentVersion from win32 where ( cver != pver and cver != '');"
$ sqlite3                 win32.db "update win32 SET pver = cver WHERE ( pver != cver and cver != '' ); "
$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad                 3.6         1         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0

Running the five sqlite3 commands another four times will bump the error counter to 5 and it should look like this:

sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad                 3.6         5         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0

The next time the 5 commands are run and current version is still an empty string, the following should happen:

$ sqlite3 -header -column win32.db "select program AS ProgramName,err AS ErrorRuns from win32 where err >= 5;"
ProgramName  ErrorRuns 
-----------  ----------
Metapad      5         
$ sqlite3                 win32.db "update win32 set err = err  + 1 WHERE cver='';"
$ sqlite3 -header -column win32.db "select program AS ProgramName,pver AS OldVersion,cver AS CurrentVersion from win32 where ( cver != pver and cver != '');"
$ sqlite3                 win32.db "update win32 SET pver = cver WHERE ( pver != cver and cver != '' ); "
$ sqlite3 -header -column win32.db "select * from win32;"  
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad                 3.6         6         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0

The error counter did not get reset but the program was shown where obtaining the version has failed.
Manually setting current version again to simulate that the next run of the version check for Metapad was successful:

$ sqlite3 win32.db "update win32 set cver='3.6' where program = 'Metapad';"
$ sqlite3 -header -column win32.db "select * from win32;"
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad     3.6         3.6         6         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0         
$ sqlite3                 win32.db "update win32 set err = 0 WHERE cver != '';"
$ sqlite3 -header -column win32.db "select program AS ProgramName,err AS ErrorRuns from win32 where err >= 5;"
$ sqlite3                 win32.db "update win32 set err = err  + 1 WHERE cver='';"
$ sqlite3 -header -column win32.db "select program AS ProgramName,pver AS OldVersion,cver AS CurrentVersion from win32 where ( cver != pver and cver != '');"
$ sqlite3                 win32.db "update win32 SET pver = cver WHERE ( pver != cver and cver != '' ); "
$ sqlite3 -header -column win32.db "select * from win32;"    
program     cver        pver        err       
----------  ----------  ----------  ----------
Metapad     3.6         3.6         0         
Pidgin      2.9.0       2.9.0       0         
Adobe Flas  10.3.181.3  10.3.181.3  0

Now you should have a utility that can check for versions and tell you if new versions are available every time it is run.
What you can do from there is left to the imagination of the reader :)

Configure a WebDAV enabled webserver for multiple user folders and one shared folder

hexeract — Fri, 25 Feb 2011 10:11:39 +0000

I recently was requested to set up an apache webserver to provide WebDAV folders for multiple users with individual folders. Additionally, all users should be able to use a shared WebDAV folder. After some extensive research I was unable to find any good hints on how to actually do this.

My first approach was to set up the WebDAV folders within the DocumentRoot and hence only one configuration item was needed with multiple configuration items. At first sight, this seemed to provide what was requested.

After some testing it showed this approach had a major security issue: If the user just accessed http://webdav.example.com and authenticated successfully, the user was able to see and write to all available folders. This is obviously an undesirable behaviour.

So I decided to move the WebDAV folders out of the DocumentRoot and providing an Alias, and configuration item for each folder and setting up access to that folder in the configuration item. Additionaly, this frees up the http://webdav.example.com which can provide further information on how to use the service.

And voilà, every user has his own WebDAV folder and can not see or access the folders of other users.

In the following example, three WebDAV folders are configured, one for each user and a shared folder for all users.

DAVLockDB /serv/webdav.example.org/auth/DAVLock
DAVMinTimeout 180 

NameVirtualHost 10.1.1.1


    ServerName  webdav.example.org
    ServerAdmin webmaster@example.org

    DocumentRoot /serv/webdav.example.org/htdocs/

    LogLevel warn 

    ErrorLog /serv/webdav.example.org/logs/error.log
    CustomLog /serv/webdav.example.org/logs/access.log combined

    # user1
    Alias /user1 /serv/webdav.example.org/webdav/user1

    
        DAV             On
        AuthType        Basic 
        AuthName        "My WebDav Directory"
        AuthUserFile    /serv/webdav.example.org/auth/webdav.user
        Require         valid-user 
    

    
        Require     user user1
    

    # user2
    Alias /user2 /serv/webdav.example.org/webdav/user2

    
        DAV             On
        AuthType        Basic 
        AuthName        "My WebDav Directory"
        AuthUserFile    /serv/webdav.example.org/auth/webdav.user
        Require         valid-user 
    

    
        Require     user user2 
    

    # transfer 
    Alias /transfer /serv/webdav.example.org/webdav/transfer

    
        DAV             On
        AuthType        Basic 
        AuthName        "My WebDav Directory"
        AuthUserFile    /serv/webdav.example.org/auth/webdav.user
        Require         valid-user 
    

    
        Require    valid-user

If you want to allow the user to access his WebDAV directory using an Internet browser you can add the following lines to the corresponding configuration item.

 Options +Indexes
 IndexIgnore ..
 IndexOptions -IconsAreLinks NameWidth=* FancyIndexing SuppressLastModified FoldersFirst 
 IndexOrderDefault Ascending Name

Dealing with a Java keystore when keytool is not enough

hexeract — Tue, 16 Nov 2010 11:51:18 +0000

It seems odd but the keytool program for handling a Java keystore is missing two rather obvious abilites.
One missing ability is that keytool is unable to import a private key and the corresponding certificate to create a Java keystore from scratch.
Another is to provide a way to extract a private key from the Java keystore.

All you need is wget, a java compiler, OpenSSL and some spare time.

ImportKey.java was written by Joachim Karrer and Jens Carlberg
ExportPriv.java was written by Alexey Zilber
Base64Coder.java was written by Christian d’Heureuse, Inventec Informatik AG, Switzerland

How to create a fresh Java keystore with a private key and a corresponding certificate

    $ openssl pkcs8 -topk8 -nocrypt -outform der -in somename.key.decrypted -out somename_pkcs8_key.der
    $ openssl x509 -inform PEM -outform DER -in somename.crt -out somename_certificate.der
    $ wget http://www.agentbob.info/agentbob/80/version/default/part/AttachmentData/data/ImportKey.java
    $ javac ImportKey.java
    $ java ImportKey somename_pkcs8_key.der somename_certificate.der

This will create a keystore somewhere (for me, on a Windows machine, the location was: C:\Documents and Settings\snowy\keystore.ImportKey ).

How to extract a private key from the Java keystore

    $ keytool -export -alias mykey -keystore mystorage.jks -file exported-der.crt
    $ openssl x509 -out exported-pem.crt -outform pem -in exported-der.crt -inform der
    $ wget http://mark.foster.cc/pub/java/ExportPriv.java
    $ wget http://www.source-code.biz/base64coder/java/Base64Coder.java.txt 
    $ mv   Base64Coder.java.txt Base64Coder.java
    $ javac ExportPriv.java Base64Coder.java
    $ java ExportPriv mystorage.jks mykey changeit > exported-pkcs8.key

How to create a PKCS#12 which IKEYMAN will accept

hexeract — Sun, 07 Nov 2010 11:56:56 +0000

This will apply if you did not use IKEYMAN to create the private key and the certificate request.
Although IKEYMAN does allow to import other key storages (another key.db, Java keystore or PKCS#12) it is not as easy as it may seem.

If you had your private key and certificate request in another key.db you would use that and not have a need to import as you could simply use that key.db file.

The Java keystore is another matter and has even more problems than IKEYMAN. The keytool program does not allow the import of private keys.

For a simple import this may seem like a lot of effort and OpenSSL does provide to create a PKCS#12 so the obvious choice would be to just do that. Once a PKCS#12 is created IKEYMAN will happily read the file, request the container password and then fail…

The reason is that IKEYMAN does accept a PKCS#12 container but has some restraints towards the encryption algorithm used for the private key and certificate.

The working command for creating a successful PKCS#12 that IKEYMAN will accept is:

 
     $ openssl pkcs12 -export -name "MyLabel" -inkey  my-key.key -in my-cert.crt -out cert-and-key.p12 -keypbe PBE-SHA1-RC2-40