Myers DS Notebook

Integrating Ruby and Linkpoint

Larry Myers — Sat, 25 Aug 2007 02:24:04 +0000

Well, after a decent summer hiatus, I’ve returned to the blog, brushed the dust off, and am ready to start writing again.

To kick things off code must be posted. A recent project involved using Linkpoint to make credit card transactions over the internet. Now yes, Active Merchant does support linkpoint as a gateway, I just couldn’t get it work right. I tried several times, futzed a bit, and eventually just wrote my own code since I wasn’t needing to do anything fancy.

Hats off to the guys behind Shopify and Active Merchant, I am very impressed with how Active Merchant has evolved, but sometimes it just ends up being easier to roll your own.

Here’s the proof of concept code I wrote that ended up giving me a “Hello World!” as far as the realm of e-commerce goes. This also demonstrates using Net::HTTPS, which might possibly be the most poorly documented part of the ruby standard library. I’d go so far to say that it is downright hidden.

Obviously I wouldn’t recommend just copy/pasting this into a production environment.

require 'net/https'
require 'rubygems'
require 'builder'

url = 'https://staging.linkpt.net:1129/'
login = '1234567890' # Your store number goes here
pem_file = File.open('1234567890.pem').read # your cert/key goes here

data = "" 
b = Builder::XmlMarkup.new(:target => data, :indent => 2)

b.instruct!
b.order do
  b.merchantinfo do
    b.configfile login
  end
  b.orderoptions do
    b.ordertype 'SALE'
  end
  b.payment do
    b.chargetotal '1.00'
  end
  b.creditcard do
    b.cardnumber '4111111111111111'
    b.cardexpmonth '1'
    b.cardexpyear '08'
  end
  b.periodic do
    b.action 'SUBMIT'
    b.installments '12'
    b.threshold '3'
    b.startdate 'immediate'
    b.periodicity 'monthly'
  end
end

puts data

uri = URI.parse(url)

http = Net::HTTP.new(uri.host, uri.port) 
http.use_ssl = true
http.verify_mode    = OpenSSL::SSL::VERIFY_NONE
http.cert = OpenSSL::X509::Certificate.new(pem_file)
http.key = OpenSSL::PKey::RSA.new(pem_file)

response = http.post(uri.path, data)

puts response.body

How to Guarantee a Code Rewrite

Larry Myers — Wed, 27 Jun 2007 12:42:46 +0000

Like everything else, code tends to move toward increasing entropy. But software developers tend to like things nice and orderly, because being able to maintain your code is a good thing. Software (unlike hardware) rarely lives in a sealed box, and will mostly like be patched, upgraded, changed frequently. Unfortunately most of these changes are made as quick hacks due to the nature of people wanting things yesterday, or the day before that if at all possible.

After a certain period your code will make you queasy, and when at that point the overwhelming urge to rewrite the whole thing becomes almost impossible to ignore.

Everybody loves a rewrite. You get to correct past mistakes, do things “the right way”, and finally get to attack all those warts that were the result of that last quick fix before the latest release.

(Chad Fowler has written several pieces about the inherent danger of the rewrite.)

How to Prevent a Rewrite

As a developer your goal should be to prevent the rewrite at all costs. It may seem like fun at first, it will become painful, especially if you have a tight deadline.

So how do you prevent your code from being tossed and rewritten? It’s easy.

Write readable code.

Really, it’s as simple as that. Your code should have plenty of whitespace, decent layout, comments … you know, all that stuff you know you should do, but never really do. If your code isn’t readable then it’s only a matter of time until it gets yanked out of the repository and rewritten (many times by yourself).

No matter how slow or buggy your code is, as long as people can understand what in the world is going on there’s at least a fighting chance that a much easier refactoring will happen instead of a complete rewrite.

It doesn’t matter what language you are using, even verbose monsters like Java can result in well written code that be read cleanly. So please, write readable code, because someone is going to have to look at it one of these days.

That Old British Guy

Larry Myers — Fri, 01 Jun 2007 11:29:19 +0000

“Broadly speaking, the short words are the best, and the old words best of all.”

- Winston Churchill

It seems like the topic of less code being better has started to spring up again. Though I do like the Wil Shipley’s spin on the meme about making your code inflexible. I figure it’s only a matter of time before people start blogging again about writing code to be readable.

This is why Ruby is a fantastic language. It is one of the only languages I’ve ever used where I can write code, leave it alone for 6 months, and still have an idea of what’s going on when I come back to it. Ruby may not be as fast as C, or have a VM like Java and .NET, but it resists code rot far better than any other language I’ve ever used.

Investigating RubyWorks

Larry Myers — Tue, 29 May 2007 00:56:36 +0000

I was reading on DHH’s blog that more than 40% of ThoughtWorks new projects are in Ruby. This is exciting, because getting both Ruby and Rails into big business environments will go a long way towards both the legitimacy of Ruby and helping ease the pain of being stuck in a Java monoculture.

Digging around the ThoughtWorks site led me to their RubyWorks page and the associated rubyforge project. My first reaction was pure excitement. In a nutshell RubyWorks is Ruby, Rails, Mongrel, and a few other tools bundled into a single package, with some magic to configure everything out of the box.

Supplying a supported one-click install for RHEL and Cent OS is exactly what is needed to help get Ruby into big business. If you’ve never worked in an “Enterprise” environment before, it has everything to do with support contracts and using software that been around long enough that all the gotchas and pitfalls are publicly known.

(Believe it or not, being “Enterprise ready” rarely has anything to do with performance and well written software. I’ve seen J2EE web apps in production that are so far away from good programming practices that the mind boggles.)

I’m excited to see what the ThoughtWorks guys can do with Ruby, Rails, and JRuby this summer. The faster Ruby gets established in big business the better.

DBI and Very Large Data Sets

Larry Myers — Fri, 25 May 2007 04:10:01 +0000

Recently I had to do some post processing on a MySQL table with 10+ million rows of data. The amount of logic involved necessitated the use of a script, and stored procedures weren’t an option. The problem was that storing that many rows in memory really isn’t feasible, unless you’re blessed with several gigs of memory just sitting around unused.

Also, when using a garbage collected language (Ruby, Python, Perl … or if you must, Java), you just don’t have as much control over your memory usage, so the problem of keeping memory allocation under control is compounded.

Two Options (That I Couldn’t Use)

1. Cursors

If you are using a database that supports cursors, this is a great time to use them. Buffer the data to your script in a reasonably sized chunk. Problem solved.

2. mysql_use_result

Rather than reading all the data at once to a local buffer, you can use mysql_use_result to read the data directly from the database as you need it.

This wasn’t an option for me since it’s a blocking process and I needed to write the updated rows back to the table as they were processed.

Paging, with a Twist

Update: The solution below stands true for Perl DBI/DBD-mysql. When using Mysql/Ruby’s free method on Mysql::Result objects it frees up memory just fine. No need to use two scripts when paging.

My first instinct was to use LIMIT and OFFSET to page through the data in reasonably sized chunks. Yes, it does hurt to have the sort the table every time before you can return data, and this method isn’t terribly fast when your offset gets big. But it does work.

The problem is that reusing the same statement handle over and over again was that the script still eating a ton of memory during the process (which was rather long running, several hours minimum). The memory would never be released until the script finally exited. My research led me to the answer that this is just an unfortunate side effect of the way most garbage collectors work.

The solution? Have the script that does the actual processing work run separate from the script that determines the offset for each pass through the loop. What you end up with is a script that invokes the processing script as a separate instance for each page of data returned from the database. As each child process exits the memory is released back to the OS.

Some pseudo code (should be similar logic for Perl, Ruby, and Python):

The parent script:

total_rows = statement.exec( SELECT COUNT(*) FROM table )
limit = 100000
offset = 0

while ( offset <= total_rows )
  system(processor_script limit offset)

  offset += limit
end

And the child script:

chunk = statement.exec( SELECT * FROM table ORDER BY id LIMIT limit OFFSET offset )

for each row in chunk
  do some processing logic here
end

It still kind of strikes me as a hack, but it works. The memory usage of each invocation of the script stayed well under 50 mb of memory, which was a big difference from the several hundred mb of memory the script was previously eating up.

Dabbling in Python and Perl

Larry Myers — Wed, 23 May 2007 23:57:48 +0000

The past few months have given me the chance to explore both Python and Perl a bit. I’m using both languages to generate simple scripts, nothing web related yet. Since I’ve been working almost exclusively with Ruby for the past few years it’s been nice to get a different perspective when it comes to dynamic languages.

Python

First off, using whitespace instead of brackets or begin .. end didn’t bother me at all. People who dismiss Python because of the indentation rules need to get over themselves. I was also very impressed with the overall polish of the docs and libraries. Python definitely makes it easy to get things done, and never had to look hard for a library and documentation to accomplish a task. I especially appreciate that MySQLdb for Python emulates cursors in ways that I’d expect.

One of the things that bothered me was that Python can’t make up it’s mind when it comes to functions. Most string functions are Ruby style (i.e. “hello”.strip(‘ell’) ), but then there’s weird exceptions like len(). Why you would have len(“hello”) instead of “hello”.len() is beyond me.

Overall though I came away with a good impression of Python, and think Ruby could learn something from the quality of the documentation and libraries. I’m hoping this will come naturally as the Ruby community continues to grow.

Perl

I understand why people tend to rip apart Perl when given the chance. Yes, yes, I realize Perl has CPAN, and it has about every library under the sun, but I haven’t found it a good enough excuse to voluntarily use the language when given a choice. The syntax made my head hurt, and once I wrote more than a few hundred lines of code I found it hard to keep my train of thought as I worked through a program.

The one thing that struck me as odd about Perl, yet I kind of liked, what that for many of its functions it just used the syntax of many of its unix equivalents. If you know sed then you know how to do string replacement in Perl.

A Recommendation

Perl programmers, please, please move to Ruby. You will find yourself happier and with much more readable code. You will also get a language where object oriented features don’t feel tacked on and half finished.

How To Build JRuby on OS X

Larry Myers — Fri, 11 May 2007 02:21:39 +0000

The JRuby has been getting a lot of press lately. It’s got support in NetBeans, and is getting a lot of support from Sun. It looks like it’s a popular topic at this year’s JavaOne as well.

JRuby is exciting because it gets Ruby a backdoor to the corporate world. Tomcat and Java 1.5 are common enough in big business these days, and with JRuby nearing 1.0 we’ll soon have a way to write Ruby code and not have to ruffle the feathers of sys admins with new dependencies for deployment.

Here’s how to get JRuby up and running on your Mac.

What You’ll Need

Download JRuby

First you’ll need to download the JRuby source, since we’ll be building it ourselves. You can get the current release from the JRuby Homepage. I used version 0.9.9 when writing this.

Install the Developer Tools

If you haven’t already, you’ll need to install the OS X Developer Tools. You can find them on the install DVDs that came with your Mac. This gets us Ant, which we’ll need later. If you already have Ant installed from somewhere else, just make sure it’s version 1.6.2 or greater (the Developer Tools version is 1.6.5).

Set Your Environment Variables

Add these two lines to your ~/.bash_profile:

export JAVA_HOME='/System/Library/Frameworks/JavaVM.framework/Home'
export ANT_HOME='/Developer/Java/ant'

Build JRuby and Run the Tests

Now you should have your build environment all set and ready to go. First extract the source files and go into the directory that is created:

tar xvzf jruby-src-0.9.9.tar.gz
cd jruby-0.9.9

Copy the provided JUnit lib over to Ant:

cp lib/junit.jar /Developer/Java/ant/lib/.

And finally:

ant test

You should see a bunch of output, and after about a minute or so you should see all the included tests (hopefully) pass. Now you can run one of the samples just as a sanity check:

bin/jruby samples/fib.rb

And there you have it, JRuby on OS X ready to go.

Code Bloat Without Trying

Larry Myers — Thu, 10 May 2007 20:39:34 +0000

Which one of these four was designed by a committee with a fluffed up ego?

(Go look at this brief post on Hackety, really, go ahead, I’ll wait.)

Ladies and gentlemen, I give you Silverlight! Guaranteed to make code unintelligible and absolutely requiring an IDE to generate most of it for you!

Tcl/Tk has been around forever (at least in internet terms), and you can tell that it was designed for people that want to create things, not wallow in schemas and brackets.

I think Microsoft might have missed the boat on this one, no matter how whiz-bang-cool the technology is.

Performance Gains using MySQL Full-Text Searches

Larry Myers — Wed, 09 May 2007 01:16:28 +0000

A recent project required me to do searching on some text fields stored in a MySQL database. In the past I’ve used statements like this:

SELECT name FROM people WHERE name LIKE '%bob jones%';

Now this works fine for smaller sets of data, but when you suddenly have data sets with millions of records it no longer works (at least if you remotely care about performance). A normal index won’t do much good either, since MySQL won’t use an index if your search string has a leading wildcard.

The solution? MySQL’s full-text search.

First thing to do is to add the index to the table:

ALTER TABLE people ADD FULLTEXT INDEX name_index(name);

Now you can search against the table like this:

SELECT name FROM people WHERE MATCH(name) AGAINST ('"bob jones"' IN BOOLEAN MODE);

Now I’m using boolean searches to get an exact match on the name “bob jones”, but you can drop the boolean mode and you’ll get search results based off relevancy. MySQL actually has some pretty good documentation on everything you can do with full-text searches.

In my research I was seeing incredible performance improvements, I had queries which were previously taking 250-275 seconds reduced to 0.5 seconds.

Also, as always, be aware that your inserts will take longer since the table will have to regenerate indexes with every insert. The MySQL docs has a good section on how to increase the speed of your inserts to minimize the hit you take from using indexes.

Simplist 2 For Public Consumption

Larry Myers — Mon, 23 Apr 2007 22:24:16 +0000

A quick note for those of you read regularly, I’ve stripped out the Myers DS specific code of Simplist 2 and made it available for anyone who wants to try it out. It requires a little leg work to get it up and running, just like any Rails app, but the migrations get you up and running right away and there’s very generic design to start with.

Simplist CMS Download Page

If you missed the announcement, I’ve rewritten the CMS that powers this site on Rails 1.2. The code is cleaner, the interface is easier to work with, and I feel it’s a substantial improvement over the original version. Check it out if you’re looking for a CMS that works with you and serves a good base for building a Rails app that needs content management functionality.

Feel free to contact me with any questions, comments, or bugs you find.