Simone Carletti's Blog » rubywhois

Whois 2.0

Simone Carletti — Thu, 17 Mar 2011 20:58:19 +0000

One year after the first major release, almost 2 years after the first commit, 1k commits and 59 versions after, I’m very proud to announce the immediate availability of Whois 2.0.

Whois is an intelligent, pure Ruby, WHOIS client and parser. It provides a flexible and programmable API to query WHOIS servers and look up IP, TLD, and domain WHOIS information. It also offers command-line interface to run WHOIS queries from the console.

This second major release, the 60th since I released the gem, is a very important milestones. I started working on this project with the idea of creating a Ruby parser for each TLD available and it took me two years and countless hours to reach this goal.

Some stats

To give you a rough idea about the complexity of this project, here’s a few numbers about Whois 2.0.

866,706 different WHOIS records (~3.3 GB of data) analyzed as of March 16th, 2011
425 RSpec test files
444 fixture files extracted from real WHOIS records
3787 RSpec examples and an unknown number of matches (sorry, I stopped keeping track of them when they reached 15k assertions) that probably makes this project one of the gems with the largest RSpec test suite
~20.000 lines of code, including the test files
168 different WHOIS parsers

Oh, man. You can’t believe the amount of work required to cover all the existing WHOIS servers, their features and especially their whims.

New and Notable

The increasing support for all existing TLD servers is probably one of the most important changes for this milestone. But the CHANGELOG is very huge and there are a few other changes that are worth a mention.

I already blogged about a few of them.

The #nameservers property now returns a Nameserver object. This change allows the parser to also extract name server IPv4 and IPv6 addresses.
A new caching mechanism reduces the number of instance variables created.
The library now supports the Gem Testers project.
You can now bind a different address or port to the WHOIS socket request.

Last but not least, the Whois::Answer class has been renamed to Whois::Record.

There are a couple of backwards-incompatible changes to the API. These changes were necessary to evolve the API to better reflect the representation of the existing WHOIS responses.

What’s Next?

The Whois library is pretty stable. It is now used in production for several big projects.

The API is mature. A few changes have been made in this major release to support additional WHOIS properties, however I don’t expect big changes in the future.

The following releases will focus on increasing the list of supported properties for each parser, as well closing some of the existing issues.

The biggest change in the API in the near future is likely to be the standardization of the status property.

In the last months I received some great feedback from people using my library with other platforms, such as Java and .NET. This is amazing.

If you have any case study, case history or feedback, please let me know. I would love to learn more about how you use the library.

How to Upgrade

To upgrade to Whois 2.0 please follow the instructions in the documentation page.

New in Whois: #nameservers are now Nameserver

Simone Carletti — Mon, 07 Mar 2011 08:58:51 +0000

It’s time for another update on Ruby Whois library. Let’s talk about name servers.

The most part of WHOIS servers follow the “convention” to include the name server list of a domain in the WHOIS record.

A name server is represented by a DNS name, which is a string.

1 2	ns1.google.com ns2.google.com

The DNS name resolves to an IPv4 IP address.

1 2	ns1.google.com => 216.239.32.10 ns2.google.com => 216.239.34.10

Optionally, the DNS name can resolve to an IPv6 IP address.git

1 2	ns1.nic.fr => 2001:660:3003:2::4:1 ns2.nic.fr => 2001:660:3005:1::1:2

Each WHOIS server arbitrary choses the amount of information to store and return in a WHOIS record. Some registries returns only the DNS name, other registries include IP addresses as well.

In the current WHOIS version, the list of name servers is accessible through the #nameservers property.

1
2
3

r = Whois.query "google.com"
r.nameservers
# => ["ns1.google.com", "ns2.google.com"]

The method returns an Array of String. This is a lossy implementation because it completely ignores the IPv4 and IPv6 details.

The new Whois version solves this problem by introducing a new Whois::Answer::Nameserver object. This object is a Struct composed by three attributes: name, ipv4 and ipv6.

1
2
3
4
5
6
7
8
9
10
11
12

ns = Nameserver.new(
:name => "ns1.nic.fr",
:ipv4 => "192.134.4.1",
:ipv6 => "2001:660:3003:2::4:1"
)

ns.name
# => "ns1.nic.fr"
ns.ipv4
# => "192.134.4.1"
ns.ipv6
# => "2001:660:3003:2::4:1"

In the new Whois version, the #nameservers implementation has been changed to return an array of Nameserver instead of a simple String.

1
2
3
4
5
6
7

r = Whois.query "nic.fr"
r.nameservers
# => [
# #,
# #,
# ...
# ]

It’s important to know that this change can break backwards compatibility. In order to minimize the impact, Nameserver#to_s returns the String value of the name attribute.

1 2	Nameserver.new(:name => "ns1.nic.fr", :ipv4 => "192.134.4.1").to_s # => "ns1.nic.fr"

New in Whois: improved caching

Simone Carletti — Tue, 15 Feb 2011 11:15:45 +0000

I’m working very hard to include some of the most important features in the new version of the Ruby Whois library.

Today, I’m very happy to report that this week I closed the issue #18 which introduces a completely new caching system for the Whois::Answer::Parser.

The way Whois parsers currently work, is to extract a property only the very first time it is requested.

1
2
3
4
5
6
7
8
9
10
11

r = Whois.query "weppos.it"

# the property has never been requested before
# the value is computed, cached and returned
r.status
# => :ok

# the property has been requested before
# the value is returned without further elaborations
r.status
# => :ok

So far, so good.

The way the system works under the hood, is to create a parser instance variable for every single requested property.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

r = Whois.query "weppos.it"

# get the first parser
# because
# r.status relies on
# a
# r.parser.parsers.first.status
p = r.parser.parsers.first

# value is not cached
p.instance_variable_get("@status")
# => nil

p.status
# => :ok

# value is cached
p.instance_variable_get("@status")
# => :ok

So far, quite good. This approach has a couple of drawbacks.

First, it creates an instance variable for every single property. Because of the large (and increasing) number of properties, the parser object space counts a large number of instance variables. This makes it hard, for instance, to sweep the cache because you have to loop through all instance variables and remove each one.

Second, there’s a small inefficiency here. Because in Ruby you don’t have to define variables, an undefined instance variable is nil. But nil is actually a value and properties can have a nil value. In this implementation, you don’t have a way to distinguish when a value is nil and when it hasn’t been elaborated yet, thus nil properties will never hit the cache.

1
2
3
4
5
6
7
8

def created_on
@created_on ||= if very_expensive_scan(/created_on/)
# ... set the value
end
end

# calling #created_on several times will continue to perform
# the very_expensive_scan as long as the value != nil.

The new approach uses a single instance variable called @cached_properties as cache. The variable contains a Hash<:key => value>, where the key is the property and the value the cached result.

If the cache doesn’t contain any key for given property, then the method hasn’t been executed yet. Cached nil properties will return nil without further elaboration.

The method #cached_properties_fetch takes care of everything.

1
2
3
4
5
6
7
8
9
10
11
12
13

def created_on
cached_properties_fetch(:created_on) do
nil
end
end

# value is cached and returned
created_on
# => nil

# the request hits the cache
created_on
# => nil

If you need to sweep the cache, reset @cached_properties to an empty Hash.

New in Whois: Gem Testers

Simone Carletti — Mon, 14 Feb 2011 16:18:37 +0000

At the beginning of February, Engine Yard announced a very interesting project called Gem Testers.

Gem Testers was built to make it easier for gem developers to write gems that run everywhere. This is split into two pieces:

rubygems-test is a Rubygems plugin which automatically runs the test suite of a specified gem on install.

Gem-testers.org is a central repository for test data. Test results are organized primarily by Ruby version and operating system.

I believe this is a wonderful project and it can have a very good impact on the quality of the available Ruby gems.

Whois currently counts 211 RSpec examples and 1899 tests with 8674 assertions. This is a very huge test suite which is able to provide an immediate feedback on existing bugs.

The new Whois version now includes support for gem test. You can decide to run the tests automatically when installing the gem or run them later by executing

1	$ gem test whois

Read rubygems-test README for further information.

Unfortunately, this feature forced me to re-include all the test fixtures and files into the packaged .gem. It means the size of .gem file will be bigger than previous releases.

New in Whois: Goodbye RDoc, Welcome Yardoc

Simone Carletti — Fri, 10 Dec 2010 09:08:35 +0000

YARD is a documentation generation tool for the Ruby programming language.  It is a very powerful alternative to RDoc, actively developed and maintained by Loren Segal.

YARD allows you to use Markdown or Textile to write your documentations, provides structured tags to better describe your methods and has a very advanced parsing mechanism. It also under active development, which is a very good sign of interest.

Back in September I asked for feedback about using Yardoc over Rdoc and YARD won with full consensus.

I’m now proud to say that the master branch of the Whois library is now entirely documented with YARD. It took several weeks, but I’m really satisfied with the result. I also had the chance to fix several documentation errors an increase documentation coverage.

The result of Yardoc for Whois README

The Yardoc documentation will eventually replace the current Whois RDoc documentation. I’m sure that it will be an awesome integration to the existing Whois manual.

In the meantime, you can try it yourself by downloading the library from Github and using the YARD server, an other awesome YARD feature.

1
2
3
4

$ gem install yard
$ git clone https://github.com/weppos/whois.git
$ cd whois
$ yard server --reload

The result of Yardoc for Whois::Answer

I already converted the Public Suffix Service API documentation to YARD a few weeks ago.

Ruby Whois 1.0 is here!

Simone Carletti — Sat, 06 Feb 2010 12:54:32 +0000

Whois is an intelligent pure Ruby WHOIS client and parser.

It is a OS-independent library and doesn’t require external C libraries or Gems: it is a 100% Ruby software with all the advantages and disadvantages that it involves. This software was developed to power RoboDomain and it eventually become a standalone library.

Whois provides the following key features:

Ability to query registry data for IPv4, IPv6, TLDs, and domain names
Ability to parse WHOIS responses
Flexible and extensible interface (e.g. You can define custom servers on the fly)
Object oriented design, featuring 10 different design patterns
Pure Ruby library, without any external dependency other than Ruby itself
Compatible with Ruby 1.8.6 and greater, including Ruby 1.9 branch
Successfully tested against several Ruby implementations, including Ruby, Ruby Enterprise Edition, JRuby, and MacRuby

Whois 1.0

Whois 1.0 is the first major release since I started working on the library one year ago. It reached an high level of maturity and stability and you can safely consider it production ready.

Compared with previous version, Whois 1.0 includes a really huge CHANGELOG (more than 100 lines!). The most important features are:

Updates to existing definitions
48 New WHOIS Parsers
Ability to check whether a property is set
Command Line Improvements
Changes to the internal Answer/Parser architecture to improve usability

Last but not least, a dedicated site is now available at ruby-whois.org.

A Retrospective

Whois 1.0 is a really important milestone. The first Whois release was just another WHOIS library. Yes, it offered support for all existing TLDs, but that was just the beginning. My idea was to creare a library to be able to parse and access all WHOIS responses with a single, standardized, object oriented API.

The current Whois library perfectly fit this milestone. Unfortunately, there’s still a big number of unsupported server, but I’m working really hard to reduce the gap. Also, in the last months a small number of users contributed back with servers, patches, changes and I’m really grateful to them.

If you look at the CHANGELOG, Whois has improved day after day.

The Client side

Whois 1.0 is a pure Ruby WHOIS client. It means you can install and use it on any platform where Ruby is supported, including (but not limited to) Microsoft Windows, Linux, MacOSX and Solaris.

This is one of the most important feature, compared with other WHOIS libraries. Whois is shipped with a Command Line Interface you can use to lookup a domain.

1	$ ruby-whois google.com

And this works on Windows too!

Whois is an intelligent Whois client. It means, you don’t need to know anything about WHOIS, Dns or TLD. Simply feed the client with the domain you want to lookup, and wait for the response.

1
2
3
4
5

require 'rubygems'
require 'whois'

a = Whois.query "google.com"
puts a

Whois will automatically choose the best WHOIS server according to the given query. If the query is a domain, Whois will try to find a server for the corresponding TLD. If the query is an IPv4 or IPv6, Whois will send the request to the corresponding management authority.

TLD and IP definitions are included in the package and read when the library is loaded. You can always update/change them at runtime.

The Parser side

Until now, Whois just seems to be an other powerful-but-standard WHOIS client. But this is not true.

The biggest difference between Whois and other WHOIS libraries, is that Whois can parse and decompose a raw WHOIS response into a powerful Ruby object.
This means, you can access WHOIS property calling a simple method rather than dealing with complex regular expressions in order to extract the data you need.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

require 'rubygems'
require 'whois'

a = Whois.query "google.it"
puts a.available?
# => false

a.nameservers.each do |nameserver|
puts nameserver
end
# => ns1.google.com
# => ns4.google.com
# => ns2.google.com
# => ns3.google.com

puts a.created_on
# => Fri Dec 10 00:00:00 +0100 1999
puts a.expires_on
# => Sat Nov 27 00:00:00 +0100 2010

You can read more at ruby-whois.org.

RoboDomain

Whois library has been creates to power RoboDomain project. With Robodomain you can keep track of any domain in one single place: log transactions, store notes, check domain status and enjoy our network tools.

Be sure to signup for an invitation code if you want to join the beta program.

Contribute

Whois is an open source project and released under the terms of the MIT license.

You can contribute by suggesting new features and reporting bugs or you can fork away the project and submit your changes.

Do you have a question? Need an help with the library? Join the discussion group.

Enjoy Ruby Whois 1.0.

1	$ gem install whois

New in Whois: command line improvements

Simone Carletti — Thu, 04 Feb 2010 20:18:26 +0000

Whois 0.8.0 was released with a new command line tool to perform one-line WHOIS queries.

1	$ ruby-whois google.com

WHOIS requests can fail and, unfortunately, the existing version of the CLI doesn’t handle this case very well. When an exception is raised, the CLI exits printing an unfriendly Ruby stack trace.

1
2
3
4
5
6
7
8
9
10
11
12

$ ruby-whois karnaugh.za.net
/Users/weppos/.rvm/ruby-1.8.7-p174/lib/ruby/1.8/timeout.rb:60:in `ask_the_socket': execution expired (Timeout::Error)
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/server/adapters/base.rb:77:in `query_the_socket'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/server/adapters/standard.rb:34:in `request'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/server/adapters/base.rb:52:in `query'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/server/adapters/base.rb:66:in `with_buffer'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/server/adapters/base.rb:51:in `query'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/client.rb:72:in `query'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/lib/whois/client.rb:70:in `query'
from /Users/weppos/.rvm/gems/ruby/1.8.7/gems/whois-0.8.1/bin/ruby-whois:37
from /Users/weppos/.rvm/gems/ruby/1.8.7/bin/ruby-whois:19:in `load'
from /Users/weppos/.rvm/gems/ruby/1.8.7/bin/ruby-whois:19

I’m happy to say that now his is just a bad memory. In case of error, the new CLI writes the error message to STDERR then terminates the execution and sets the exit status to 1.

1
2
3
4
5

$ ruby-whois karnaugh.za.net
Request Timeout

$ ruby-whois google.es
This TLD has no whois server, but you can access the whois database at `https://www.nic.es/'

Ruby Whois moved to ruby-whois.org

Simone Carletti — Wed, 03 Feb 2010 19:08:44 +0000

The Whois 1.0 release is just around the corner and to celebrate the first major release the library is changing home from my Codestuff site to ruby-whois.org.

The repository is still hosted on GitHub and in the next days I’ll probably migrate the issue management as well.

The design of the new Whois website is clearly influenced by this website and Rack::Cache. All HTML files are automatically generated using Jekyll. The new site contains all the existing documentation published on the legacy wiki, reorganized and updated. As soon as Whois 1.0 will be available, I promise to spend some time writing more details about the advanced features such as creating WHOIS parsers and using the answer object.

New in Whois: property is set?

Simone Carletti — Fri, 22 Jan 2010 22:46:21 +0000

The new Ruby Whois version will offer the ability to check whether a property is set using the standard method? Ruby convention. This means, you no longer need to check whether a property is supported, get the property value and compare it to nil.

1
2
3
4
5
6
7

a = Whois.query "google.it"
a.created_on?
# => true

a = Whois.query "notregistered.it"
a.created_on?
# => false

Two important aspects to keep in mind. First, this method works only at answer level.

1
2
3
4
5

a = Whois.query "google.it"
a.created_on?
# => true
a.parser.created_on?
# => NoMethodError

Second, this method doesn’t care whether the property is supported or not. It returns false either if the property is not supported or the value is nil.

1
2
3
4
5
6
7
8
9
10
11
12

a = Whois.query "notregistered.it"

# created_on is a property supported
# by .it TLD parser and empty for this
# specific query
a.created_on?
# => false

# domain_is is not a property supported
# by .it TLD parser
a.domain_id?
# => false

New in Whois: Applying the Principle of Least Surprise

Simone Carletti — Sun, 17 Jan 2010 12:28:01 +0000

In the past weeks I’ve been working to improve the Ruby Whois Gem. While most of the effort is actually focused on creating new WHOIS parsers for the existing registries which are more than 400, I also took the time to add some new features.

One important change is strictly focused on applying the principle of least surprise in the Whois::Answer design. Prior the changes, calling a method on a Whois::Answer instance to read a property value might result in 3 different behaviors:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# the property is supported, return the value
a = Whois.query "google.it"
a.created_on?
# => Fri Dec 10 00:00:00 +0100 1999

# the property is supported but blank, return the value
a = Whois.query "notregistered.it"
a.created_on?
# => nil

# the property is not supported, raise an exception
a = Whois.query "google.it"
a.domain_id?
# => PropertyNotSupported

Whether the parser supports a property depends on the WHOIS record. For instance, a .name WHOIS response doesn’t contain the domain expiration thus the property expires_on is not supported. Contrariwise, the .it WHOIS response contains that value thus you can safely call expires_on to get the property value.

How do you deal with supported/unsupported properties? With the old Whois version you would have to check whether a property is supported before actually calling the getter method.

1
2
3
4
5

# the property is supported, return the value
a = Whois.query "google.it"
if a.property_supported?(:domain_id)
a.domain_id
end

That worked well but was… horrible. Dogfooding the library working on RoboDomain I ended up with the conclusion this wasn’t the behavior an user would expect. For this reason, the new Whois version returns the value if the property is supported, nil otherwise regardless property status.

This change only affects the answer object. If for some reason you need to know whether a property is supported, you can always use the property_supported? method or access the underlying parser implementation.

1
2
3
4
5
6
7
8
9
10

a = Whois.query "google.it"

a.domain_id
# => nil

a.supported?(:domain_id)
# => false

a.parser.domain_id
# => PropertyNotSupported