Toronto Rails Night - Merb Presentation

11 June, 2008

Last night was yet another successful Toronto Ruby on Rails Nite put on by Corina Newby. The Toronto ruby community is rocking, evidenced by the large turnouts and fresh faces we see everytime. I'd guess 40-50 of Toronto's best and brightest Rails developers filled out the room. The unspace guys put on some much appreciated beer and made sure we're all informed about the upcoming RubyFringe (can't wait, I'm really hoping Zed's presentation is as exciting as his blog rants posts). Along with the other presenters of the night Mike and Libin/Wes/Carsten (up on TorontoRB.ca sometime in the next day or so). I did a presentation on Merb - The pocket rocket framework. Slides are are here (you may have to click through to the blog post to view if you're reading this in a reader/aggregator). Resource links are at the end of this post, and some discussion notes follow.. Presentation background Intended for people who are interested in looking at Merb, collating key points from blog posts, irc dicussions, mailing lists, and own experiences getting an app running with Merb (well the opposite way round, trying to get an app running and finding out those answers!). Hopefully the presentation collected a fair few hours worth of investigation into 20 or so minutes to help other people out Sports Bikes vs Sports Cars Useful analogy between Sports Bikes (Merb) and Sports Cars (Rails). Said cars have all the bells and whistles, everything is packaged up nicely, and it's pretty hard to hurt yourself, however given the nice packaging it's pretty hard to modify things. Sports bikes (Merb) on the other hand, are much more open to tuning, playing with, go faster, but need a bit more care Same same but different Rails and Merb apps are very much of the same likeness, but there are differences. Don't expect to be able to port code over 1:1 (especially if you've got a big reliance on plugins), and don't pepper the merb mailing lists/irc channel for "please make xyx behave like it is in rails merb-core & merb-more Two gems you need to get started (technically you only need merb-core). -Core is the stripped down bare minimum to get merb running. -more contains all the nice packaged goods to help you along the way merb-gen is your friend throw out script/* you're now doing everything in merb gen for generating your app, models, resources etc Key code differences refer to slides, but the code used is difference Agnostic in a very good way Merb is agnostic when it comes to ORM, test framework, js framework, rack adapator. merb-gen behaviour changes based on component used so if you're using rpsec, then stories get created vs unit tests. This is one of the reasons I'm very much into the idea of Merb Cool features part controllers, action args, router, covered off in the slides. Full apps, flat apps, very flat apps and thin servers and large grains of salt For performance stats (which are to be taken with a very large grain of salt). Hello world action on a single controller, rails would be about 300 req/s full merb app would be round 500 req/s flat merb app 500-550 req/s, a very flat app marginally quicker again, front a very flat app with thin (merb -I veryflatapp.rb -p 3000 -e production -a thin) and woah hold on, 1000 req/s ! Question was asked about benchmark figures and I made the point, they're going to be all over the place - it's easy enough to benchmark for yourself, that's the only thing that's really relevant. We know Merb is quick, just how quick will depend on your own application and uses. Merb, Rails and you The inevitable discussion at the end around, should you use Merb and how does it relate to Rails. Some of the goodness in Merb looks to be slowly being backported to Rails, so over time Rails will benefit from all of the wizardry going on in Merb (hopefully!). For those that like Merb's approach, and willing to forsake the vast library of rails plugins, and/or have very specific performance requirements and/or need to hack a framework around to do what they want. Merb looks to be killer. Ultimately it's upto you, so evaluate carefully. Resources Lots of link goodness..... http://merbivore.com http://wiki.merbivore.com http://merbunity.com http://merborials.com/ http://groups.google.com/group/merb http://github.com/search?q=merb http://mwrc2008.confreaks.com/ http://www.slideshare.net/search/slideshow?q=merb http://www.slideshare.net/search/slideshow?q=datamapper

Read More

Minor gotcha and shout out to Passenger crew

06 May, 2008

Okay so yesterday I found a behavior problem with passenger (1.04) in that

request.env['PATH_INFO']

was nil, whereas under Mongrel it was being set as we were expecting to the current path. Within less than I guess 2 hours of reporting it on #passenger, code was put in place and committed to github. Now how's that for amazing work ? You've got to hand it to the boys over at the Passenger team. I sat back in my chair at that point and thought about the days before dealing with open source. I recall a dev team I was working with found a bug in number handling with .NET, what could they do ? For all the money paid in licensing fees etc ? Nothing. Just code around it. Days like yesterday I really enjoy the open source community, it certainly has some major advantages. I still have a lot of testing to do before I'd happily switch a production app over to Passenger, but looking forward to the promise of better deployment that's for sure!

Read More

CentOS 5 + Passenger

30 April, 2008

Tip of the day: For anyone out there with CentOS 5, despite what Passenger thinks, your http-devel libraries likely are installed. I was getting an error "you need to yum install http-devel" , which I had, but still getting the error. If like me, the export APXS2 as per passenger docs doesn't work, just add "/usr/local/apache/bin/" into your profile path and passenger will then work fine (I just checked by ensuring axps ran at the command prompt before running the passenger module install) In theory then you should be good to go...

Read More

Tips of the day - Query Trace and Equality

07 March, 2008

Tip #1 Query Trace So you're profiling your application, cutting down your SQL queries, looking through the logs - "argh where did that query come from ?!" you say to yourself. Then you remember what Luke G said at the last Toronto Rails Project nite - check out Query Trace. Oh how this has made my life so much better. It tells you the stack trace at the point the query was executed, colorised nicely so you don't go blind tailing your log files. Here's a very simple example so you get the idea.. Get the plugin from here https://terralien.devguard.com/svn/projects/plugins/query_trace/README Tip #2 Object equality vs checking id's When checking for ownership or association between objects, be careful you're not checking for equality of the objects. For example "did person x create this order y" the temptation is simply to go "if person == order.creator", where you're checking the objects themselves. Careful! As this is actually loading up the creator of the order if it's not already loaded (SELECT * FROM people WHERE person.id = x). Simply check if person.id = order.creator_id, as this won't require the order's creator to be loaded, eliminating one less hit to the DB. Little things like this littered through an application can start really slowing things down if you're not careful.

Read More

How to Avoid Hanging Yourself with Rails

12 February, 2008

The law of unintended consequences. Evidently people are searching on figuring out how to leave this earth in Google and getting to this page because of the title with alarming regularity. If you're one of these people PLEASE see a psychiatrist, talk to someone you love, take a deep breath, realise that you've got your whole life ahead of you, whatever it takes. This evening I presented at the monthly TSOT Toronto Rails Project Nite. As promised here's the presentation and links to resources mentioned. Big thanks to TSOT, everyone who came, everyone else who presented and the spirit in the T. Ruby/Rails community. Presentation - How to Avoid Hanging Yourself with Rails Faker - faker.rubyforge.org 'mrj's ActiveRecord select/include patch - http://dev.rubyonrails.org/attachment/ticket/7147/init.5.rb include preloading optimisation - http://blog.codefront.net/2008/01/30/living-on-the-edge-of-rails-5-better-eager-loading-and-more/ Some points following on from this: - YMMV, times were produced using a consistent dataset, results are going to be all over the place dependent on columns, table size, indexes, ordering etc etc. - By testing with real (/faked) data, you can avoid (to some degree) the premature optimisation problem, you'll see what your app is going to perform like closer to a real world production environment. - Word on the street is Datamapper + Merb is wickedly fast, I'm going to run a test with the same dataset just to see what the difference is like - There were no MySQL optimisations preformed, just a vanilla MySQL install. - I'm not being negative on the core team about the select/include problem, it's a hard problem to fix, I'm just making people aware of it so you can save the banging-of-head-on-desk problem others of us have suffered. - If the presentation bored you to tears, you know it all already, live in Toronto, and are bored in your job - we may have a position for you. Talk to me... If you're in T. make sure you get yourself along to the Rails pub and Rails project nites, well worth it for networking and learning respectively. I hope to see every developer in T. putting forward a presentation on their projects, by collaborating and sharing we all learn immensely. Personally I've found every presentation exceedingly interesting - keep it up !

Read More

require, RMagick, and case sensitivity

19 December, 2007

(At least on Leopard with Ruby 1.8.6) Ruby will try to reload rmagick if you get the case of rmagick different from the original require. And when that happens it all goes up the whop. We were using attachment_fu's image magick processor, then doing a require 'rmagick' seperately which resulted in tonnes of errors like this

/usr/local/lib/ruby/gems/1.8/gems/rmagick-1.15.10/lib/rmagick.rb:32:
  warning: already initialized constant PercentGeometry

Keep your case's of require's consistent. Even if this is file system dependent behaviour in mixed development environments you can imagine how weird problems could get!

Read More

Automated testing of websites via “real” users.

22 August, 2007

Of all the testing we can do, there's nothing quite like firing up your browser and navigating to the website in question, logging in and doing stuff. All of the integration tests, port tests, contrived "I'm alive" tests etc can give you an *indication* that your website code is all well and good. However what you really want to test is "can I jump to it, do x y z action, and complete those actions without it bombing out". We have a two options, we hire an army of minions who will do that for us every 15minutes on the dot OR we could do it automatically. How do we do this ? Well there are a number of ways. One option I'm going to walk through is using the magic genius of Safari Watir, RSpec tests, ruby, a few gems, cron and any Mac you can lay your hands on. We set this up a while ago monitoring some of our sites and it's great for finding out when something in your webstack has fallen over. For a < $1k investment it's a no brainer for any web site. Caveat - this ain't going to test your flash/flex content - this is just for HTML based sites. Read on... 1. Machine setup First, take one mac mini. We need to install ruby, ruby gems, rails, rspec, safariwatir, and rb-applescript bridge on it. Follow the Hivelogic narrative here for instructions on installing a full rails stack. Then just simply run these commands to finish off the last remaining gems.

  sudo gem install rspec
  sudo gem install safariwatir --include-dependencies

2. First website check So we've got our infrastructure setup. Now we need to write the scripts that actually run the test the site. These will fire up safari and tell it to navigate to various URL's, then check the content on the pages returned. So in true behaviour driven development style we write a spec that in plain english walks through the steps. It fires up a browser instance courtesy of Watir::Safari.new and uses it to navigate around pages. First let's create a file called check_google.rb like so:

require 'rubygems'
require 'safariwatir'

context "With www.google.com" do
  before(:all) do
    @browser = Watir::Safari.new
  end

  it "can navigate to home page" do
    @browser.goto("http://www.google.com/")
  end

  it "homepage should not have a 404" do
    @browser.contains_text("404").should == nil
  end

  #it "should throw an contrived error, when seeing if yahoo is on main page" do
  #  @browser.contains_text("www.yahoo.com").should == true
  #end

  after(:all) do
    @browser.close
  end
end

Now if we run

spec check_google.rb

we get output as follows:

 RowanH$ spec check_google.rb
..

Finished in 2.818356 seconds

2 examples, 0 failures

Fantastic, it's what we expect google is alive and tickety-boo. Now to check the failure scenario, uncomment the yahoo test:

RowanH$ spec check_google.rb
..F

1)
'With www.google.com should throw an contrived error, when seeing if yahoo is on g main page' FAILED
expected: true,
     got: nil (using ==)
./test_mysite.rb:18:

Finished in 2.837444 seconds

Cool, we know what a failure looks like. 3. Turning your console output into a email message We're not going to be logging in and doing this all the time, so we want to be emailed of success/failures. We are going to use a custom formatter for RSpec, which can make our output look a lot more interesting and pop it into an email. As I was running these scripts on a machine with a full rails stack I used ActiveMailer for sending emails.. (you'll see why in a minute) so let's create a custom formatter that will format our output for mailing. Lets call it mail_formatter.rb and pop it into the same place as our existing scripts - as follows.

require 'rubygems'
require 'action_mailer'

#actual email thingy
ActionMailer::Base.server_settings[:address] = 'domain@myhost.com'
class Emailer < ActionMailer::Base
  def test_email(text, status)
    subject    "TEST Autotest Status - #{status}"
    from       "mytestbox@myhost.com"
    recipients "me@myhost.com"
    body       text
  end
end

class MailFormat < Spec::Runner::Formatter::BaseTextFormatter
  def start(spec_count)
    @mail_output = String.new
    @mail_output << "Test started \n"
    @mail_status = "Tests OK"
  end

  def add_behaviour(name)
    @mail_output << "\n #{name}\n"
  end

  def example_failed(example, counter, failure)
    @mail_output << " - #{example.description} FAILED \n"
    @mail_status = "#{counter} tests FAILED"
  end

  def example_passed(example)
    @mail_output << " - #{example.description} PASSED \n"
  end

  def dump_summary(duration, spec_count, failure_count, something_else)
    @mail_output << "\n"
    @mail_output << "Finished in #{duration} seconds "
    @mail_output << "#{spec_count} test#{'s' unless spec_count == 1}, #{failure_count} failure#{'s' unless failure_count == 1}"
    Emailer.deliver_test_email(@mail_output, @mail_status)
  end
end

Right so this time if we a new command like so, to require the mail_formatter, and output in mail format, we should by rights receive an email:

 RowanH$ spec check_google.rb -r mail_formatter.rb -f MailFormat

1)
'With www.google.com should throw an contrived error, when seeing if yahoo is on main page' FAILED
expected: true,
     got: nil (using ==)
./test_mysite.rb:18:

(we still get our console output if we get a failure) 5. Automating it Okay so wrap that lot up in a shell script, and call it via cron (hey I'm not going to write *everything* out here), and you've now got a simple script that's calling up your website and checking it. 6. Extending it If you browse around for some of the safari-watir examples out there then you'll see more of how to access specific fields and named form elements - this will let you actually login and do stuff on your site(s) your testing. 7. Extra Credit - Screenshots!! Okay, we've got our system emailing us every test. But what would be really cool is if it actually sent us a screenshot everytime something turned to custard and failed so we could see what actually went wrong. Well... it's suprisingly easy. We're going to use OSX's inbuilt screencapture utility and fire off a shell command to capture a png file everytime something goes wrong. Then pop those in as attachments to our status email. Modify the code like so: In the Emailer.test_mail method, add a parameter images, and pop this code at the bottom of the method:

   if ( images )
      for image_name in images
        attachment :content_type => "image/png", :filename => image_name ,
                        :body => File.read("/Users/rowanh/Desktop/failed_images/#{image_name}")
      end
    end

So this will allow our email method to read in a list of images and attach them to mail message. Next we want to update our formatter. For the start method add the following:

    @failed_image_count = 0
    @failed_images = Array.new

Add a new method like so :

  def take_screenshot()
    @failed_image_count += 1
    screenshot_name = "failure_#{@failed_image_count}.png"
    system "screencapture /Users/rowanh/Desktop/failed_images/#{screenshot_name}"
    @failed_images << screenshot_name
  end

Next update example_failed so it takes_screenshot. Finally in the deliver_test_email method, pop in the @failed_images array. Conclusion Bingo! you now have a hugely useful tester, that runs away, tests your website, tells you and will actually show you what went wrong. For not a whole lot of investment. Off to setup a continuous integration machine that actually does this for each build........ Credit: http://blog.aslakhellesoy.com/2006/12/2/getting-screenshots-from-watir http://redsquirrel.com/cgi-bin/dave/projects/watir

Read More

XChain is now running on rSpec

10 August, 2007

Whilst I wait to sort out the UI issues I'm building up the specs' for XChain. I'm using RSpec, so it's a lot more 'self documenting' about what you can expect the application to do. Starting on a fresh project, it's immensely enjoyable to get into the spec -> code -> test -> repeat cycle. If you're at all hesitant about getting into them feel free to go browse the source of my specs. I only spent a little time on it last night but already starting to see progress - a lot of the schema is getting cleaned up as a result of the specs. Here's the doc out put so far (which is about a fraction of what the finished specs will look like !).

Order when creating a new order
- should be able to instantiate
- should default to draft status
- should be able to prefill an address
- should have a default price type matching customer price type
- should calculate return 0 dollars
- should throw an error that it doesn't have line items
- should be able to be edited
- should be able to have an order line added

Order when finding existing order
- should have id = 1
- should have a purchase order number
- should have 3 order lines
- should have a billing address
- should have a customer

To get the specs running. 1. Grab a latest copy of xchain from subversion http://xchain.googlecode.com/svn/trunk/. 2. Dump all databases, then create, and run migrations. 3. Install rpsec using instructions from here 4. Run rake rspec 5. You should (at time of writing) get 13 examples with 1 failure (to work on!) By the end of the weekend there should be a healthy amount of specs in there and fair portion of the model structure of orders, pricing, customer management done. Heres a sample chunk of code

require File.dirname(__FILE__) + '/../spec_helper'


describe Order, "when creating a new order" do
  fixtures :orders, :order_lines, :customers, :addresses, :addressables, :products

  it "should be able to instantiate" do
    @order = Order.new
  end

  it "should default to draft status" do
    @order = Order.new
    @order.order_status_id.should == 10
  end

  it "should be able to prefill an address" do
    @customer = Customer.find(1)
    @order = Order.new
    @order.should respond_to(:prefill_address)
    @order.prefill_address(Customer.find(1))
    @order.billing_address.should eql(@customer.addresses.default_billing.address)
    @order.billing_city.should eql(@customer.addresses.default_billing.city)
    @order.billing_postcode.should eql(@customer.addresses.default_billing.postcode)
    @order.shipping_address.should eql(@customer.addresses.default_shipping.address)
    @order.shipping_city.should eql(@customer.addresses.default_shipping.city)
    @order.shipping_postcode.should eql(@customer.addresses.default_shipping.postcode)
  end
end

If you want an introduction on rspec follow your nose to either the rpsec docs (which are fairly self explanatory) or if you're the visual sort the head on over to http://peepcode.com/products/rspec-basics for a screencast (disclaimer, haven't watched it, but judging on Peepcode's previous efforts it should be good).

Read More

Using Rails with Flex to manage long running tasks

31 July, 2007

UPDATE Jul 2008 - The API for BackgroundRB has changed and docs improved significantly since this blog post. Read the latest API calls over here http://backgroundrb.rubyforge.org/rails/. Don't reference the Rails code below if using latest BackgroundRB As soon as you start doing anything with photos sooner or later someone says "It would be really nice to upload these in a zip file", which then leads into a whole rabbit warren of issues, one of which you will inevitably come across is how to deal with a long running task on the server, in terms of a) getting it to actually complete and b) telling the user what's going on. There are various hacks for doing these, like running a shell script and writing out tmp files all over the place. Just plain ugly. Enter Backgroundrb, probably one of the greatest little inventions for Rails. It lets you kick off a background process, outside of the main rails app process, with the option to interact with the rails models or not. The beauty of this plugin is a little thing called MiddleMan, as the name implies it lets you interact with your background processes (Workers) from your Rails app. So, this gives us the ability, from your Flex app to invoke a Worker on the server side, then monitor that Worker until it's finished, and/or kill off the worker if need be (for slacking on the job?). What do we need. 1. First off install BackgroundRB from here. 2. Next we need to create our worker in your Rails app dir 'script/generate worker ResizerWorker . Here's a simple chunk of worker code.

class ResizerWorker < BackgrounDRb::Rails
  def do_work(args)
    @total_images = 50
    @current_image = 1
    50.times do
      @current_image += 1
      sleep(1)
    end
  end

  def get_progress
    @current_image < 50 ? status = 'In progress' :  status = 'Finished'
    info = { :status => status, :current_image => @current_image, :total_images => @total_images}
    return info
  end
end

As you can see we have a do_work method which is generated by default, this is called whenever the worker is instantiated so it kicks off our work to be done. Then I added a get_progress method to return a hash of the worker's current status. 3. Great so now we need to kick it off, there's some options to do this automatically at timed intervals, but I'm not going to go into that here, all we're doing is getting our rails app, by a web request to start/stop and monitor the progress (the web request is then called from Flex) . So we create a watcher controller, with actions 'start_task', 'status' and 'stop_task' as below which is all pretty self explanatory:

class WatcherController < ApplicationController

  def start_task
    session[:worker_key] = MiddleMan.new_worker(:class => :resizer_worker)
    info = {:message => 'job_started', :key => session[:worker_key]}
    render :xml => info.to_xml(:dasherize => false)
  end

  def status
    @worker = MiddleMan.get_worker(session[:worker_key])
    if @worker
      info = @worker.get_progress
      if info[:status] == 'Finished'
        MiddleMan.kill_worker(session[:worker_key])
      end
    else
      info = { :status => 'Finished' }
    end
    render :xml => info.to_xml(:dasherize => false)
  end

  def stop_task
    @worker = MiddleMan.kill_worker(session[:worker_key])
    info = { :status => 'Task Stopped' }
    render :xml => info.to_xml(:dasherize => false)
  end

end

Gotcha! I found out the method names in the Documentation, on MiddleMan, were wrong compared to the version I had installed. I needed to use .new_worker, .get_worker and .kill_worker. 4. Fantastic, so go into console, do "rake backgroundrb:start" in your rails directory, then fire up a rails server. You should now be able to hit http://localhost:3000/watcher/start_task, http://localhost:3000/watcher/status, http://localhost:3000/watcher/stop_task and see the appropriate results. 5. Now you can fire HTTPService requests on the Flex side and consume the results as required. For example : (for the sake of berevity I haven't put in any timer function to recheck the status, just a manual click event)

	
		import mx.rpc.events.ResultEvent;
		import mx.controls.Alert;

		private function onSvcWatchTaskResult(event:ResultEvent):void
		{
			trace(event.result.valueOf());
			var info:XML = XML(event.result);
			pbProgress.setProgress( info.current_image, info.total_images );
		}

		private function onSvcStartTaskResult(event:ResultEvent):void
		{
			Alert.show('Task Started');
		}
	





	
		
		
			
			
			
		
	


And there you go. A few seemingly simple lines of code has knocked off one of the age old problems of dealing with long running tasks. I heart Rails (and Flex).

(As always with code examples, don't copy and paste, rewrite it yourself. You'll understand it better and won't copy in any errors I've made whilst cutting it down for blogging)

Read More

Open Source Rails & Flex eCommerce Application

17 July, 2007

[Announcement] Rowan Hick Consulting, Canada and Ron Hanley of Fastmount LTD, New Zealand are pleased to annouce that their new Rails/Flex/MySQL eCommerce application will be released to the community under an Open Source license. The application, is a successor to a PHP/MySQL based application developed by Rowan Hick and James McGlinn of Nerdsinc Ltd New Zealand.

After nearly 2 years in production, with the advent of Rails, Flex, and the rapidly growing business needs of Fastmount, the current application no longer meets the business needs of Fastmount. We made the decision early this year to re-write a new application and have been developing requirements for the application as well as evaluating technologies. The application is a business to business eCommerce system. Allowing a marine manufacturing company in New Zealand to service it's agents, distributors and customers world wide. The existing application facilitates order management from creation through to shipping tracking, along with customer management and some reporting elements. Why are we open sourcing ? We made the conscious decision that we don't want to sell the application, as such being closed and proprietary we gain no extra business value by keeping it internal (this doesn't preclude the possibility of a hosted version being provided down the road). The application contains no proprietary information to Fastmount Ltd so there is no harm by open sourcing it. Having done a number of eCommerce solutions, the problems faced and solutions tend to be very similar regardless of domain. Thus the app's tend to look very similar. Why reinvent the wheel, over and over again? My personal vision is that a vibrant community will flourish around the application. If it can become an option for people when putting together an eCommerce solution then I would be ecstatic. Currently I haven't seen an active Rails based open source eCommerce engine out there so there's hope that this may become one of many possible standard 'engines' for people to use. Through our background in the widget ordering domain - solving business problems beyond the simple shopping cart such as; multiple price catalogues, multiple currencies, product/ package combination, agents, sales by regions etc - this may help to be a learning tool for those who aren't familiar with the problems at hand, introducing real world solutions beyond your standard learning materials and tutorials. Also by opening it up, and allowing public scrutiny will likely ensure that a higher standard of coding is reached within the application - there's nothing like judgement from your peers to keep those nasty kludges out of the codebase. Why Flex ? Flex, does at first glance seem to be an odd coupling with Rails in terms of philosophical approach. However as a business tool, this application needs to work perfectly, regardless of browser incompatibilities. In my own opinion, the time saved by not having to deal with these issues, along the extra interface richness granted by Flex, warrants the extra upfront time in creating the Flex interface. What should be noted here is this application is more of a business to business, rather than business to consumer type system - thus we can expect less casual browsing/ordering as in the case of a product catalog/ecommerce app versus a very defined business process for ordering. Having said all this, whilst the app has a Flex front end by using some Rails niceties there will be no stopping anyone developing an HTML front end for the app (infact the app will already have some front end work in HTML), so the core Rails application will become the engine and any various front end(s) that may appear can be bolted on. We're using a REST based approach to facilitate this. Roadmap Current Status: Initial development of Order management stories; Creating, editing, listing orders and customers. Evaluating implementing microframeworks for the Flex front end (Cairngorm or Model Glue Flex). Evaluating communication methods (HTTPService, WebOrb, RubyAMF). Release 1: Production ready system - Revamp of existing system in Rails and Flex, all existing user stories maintained. Streamlined order entry system, customer management, order payment processing (via ActiveMerchant), shipment tracking (via UPS). Release 2: Customer Relationship Management enhancements Release 3: Inventory Management, manufacturing enhancements. Community Release The application will be released to the community under an Open Source license (license to be decided), the only caveat from our production system will be licensed assets (such as icons) stripped out for community release with placeholders for people to find their own. I haven't as yet investigated a trac (or trac like system) that's impervious to spam, so for any community members that have any ideas reading this post feel free to drop us a comment. This will likely happen in the next month. Community Contributions We will welcome contributions and commit back into the core, so long as they don't negatively impact core functionality. Process as yet undecided. To ensure fair attribution - the credits page of the app will include your name as a contributor. Whatever license is decided upon will require that any modifications are publicly available as open source. Share and share alike principles. The Name? Undecided - if you want to throw out an idea feel free! Keep Informed All announcements will be made here on this blog. Keep informed, keep your RSS readers locked to http://feeds.feedburner.com/rowanhick In closing I'm personally very excited about providing this to the community and hope good things will come out of it.

Read More

Syncing up external Filemaker data with Rails

28 May, 2007

One of the web apps I work on leverages a Filemaker database. My co worker Brent the expert Filemaker dude, and I had a serious problem to tackle - how to get Rails and Filemaker talking to each other. Now Filemaker has iffy SQL standard support and we didn't necessarily want our web app to run live from the Filemaker database. We spent some time looking at making an ActiveRecord adapter for FM but really at the end of the day it was too much like hard work. There are 3rd party solutions out there that can do this stuff for you, but if your needs dictate that they won't work - our experience may help you. I'm going to write up the theory behind this which has been live in production for a good 6 months + and in the coming weeks will release some code acts_as_syncable or similar... First off we wanted it occassionaly connected, one way synchronising. We had an installed instance of Filemaker Web Publishing engine so we could expose our data as xml views. We used a background rails (backgroundrb) process to query Filemaker every 5-10mins and ask for changed data, pull that data using REXML, and populate/update on the Rails side using ActiveRecord. Here's the mechanics... 1. Each object, needs a last_modified timestamp, whenever an update to the object that affects the data being synced, this last_modified timestamp needs to be updated (for all intents and purposes create an updated_at column on the filemaker side). 2. Next on the Filemaker side create a layout that exposes that data when queried via Web Publishing, as an xml result. We're going to do two different queries so your layout must support these - as per note ii. 3. On the Rails side we need to add two models, and extend the models we want synced. First I created a SyncLog model which will track when syncing occurs (just for informational purposes), then a ClassSyncLog, which will track the models synced, when they were synced, the last timestamp and id of each model synced. 4. For each model on the rails app side, we need a number of properties: - The corresponding id field name from Filemaker - A hash listing each rails field and it's corresponding name on the Filemaker side (we're assuming that these will be different, in our case they were wildly different) - Some call backs, pre process sync, and post process sync if you need to do stuff before/after syncing. 5. We need a model that will actually do the work of syncing the objects. This is the involved part of the process that does the grunt work of syncing so pay careful attention. 5.1 Pass in a list of Class names, for each class name do the following 5.2 Query filemaker using Net::HTTP from the last updated_at timestamp and see if any records have been updated since then, if so move on to 5.3 or go to the next class (well... almost there's a trick here **) 5.3 For that returned result set from Filemaker (from the HTTP Raw data returned), fill an REXML object from the returned xml file. For each of the records in the result set do the following 5.3.1 Find or create the rails side object from the corresponding id in the resultset, check if you can sync it per the pre process rules if you have any. 5.3.2 Iterate over each of the fields in the result set, see if it matches a field in the field mapping on the rails class you're syncing (4). If it does update the current object from the result set. 5.3.3 Finally save the current object and move onto the next record 5.4 Now before moving onto the next class you need to save a ClassSyncLog record, with the last timestamp of the last updated record, and the last id. So next time you have a starting point to retrieve the records. 5.5 Move onto the next class 5.6 After completing all classes write out a log entry to say it's been done. 6. Wait 5mins... and repeat. Of course there's some caveats. i. First is if you're dealing with 100'000's of records you do not want to grab just one resultset back - you're going to run out of memory and/or have problems with getting the full stream of data. So we get the result set query into groups of ~ 2000. ii. Related to this, lets say you have 100'000 records updated at 10:10:01 AM. The first request is going to be for all records greater than the last updated time, for example 9:30AM. So you'd get 2000 in on your 100'000 records, the next time in you ask for > 10:10:01 am, you're going to skip 98'000 records, so you want to do two queries first: a. Give me all records time = last_updated_time AND id > last_updated.id, which will catch the next 2000 records - repeat this one until you get no more results then: b. Give me all records time > last_updated_time - then go back to qry #a for the next repeat. iii. You will have performance issues, this isn't quick by any stretch of the imagination so be a little patient to work through the issues. We're running massive datasets so we often see updates that take hours. Not great but it's ok. iv. Watch your memory, this can be a long running process so throw in 'GC.start' to clean up memory along the way. (eg every resultset retrieved) v. To avoid ID collissions it's strongly recommended you use the MySQL master-master auto-increment-increment + offset scheme, so you stagger your ID inserts between your Filemaker DB(s) and MySQL DB(s). Again look for code in the coming weeks.. I'm wrapping our custom stuff up into an acts_as module for the (limited) audience this may help...

Read More