Our client Viscape was cited in a USA Today article on travel to DC for the upcoming inauguration as showing a 25x increase in rental listings. Viscape provides vacation rental listings and travel destination planning for property owners and travelers who want to use the wisdom of crowds to maximize their vacation experience.
Author Archive for alon
The San Francisco of Modern Art released the redesign of www.sfmoma.org on Monday. This redesign has been in the works for a couple years and is the result of a lot of hard work by the folks at SFMOMA, Hot Studio and of course Carbon Five. Congratulations to all involved.
SFMOMA.org is the latest site to be released using SmileMaker, our favorite web CMS.
Read more about this effort on the SFMOMA site.
TechCrunch is spreading the word about our client The Mechanical Zoo and their social search product Aardvark. I’m sure they’re helping the network grow. I certainly got a number of requests for invites to the beta as soon as the post was up.
We worked closely with TMZ to bootstrap the development of Aardvark. They are a fun and talented team working on a great product.
Or course we like the comment by one reader:
Technically it works surprisingly well for a new app in beta - they must have superior techies working on the details, not just marketing and sales guys - I am impressed so far and they are only at the beginning of the zoo alphabet (aa)!!!
We’re spending a couple days next week with developers at one of our clients to practice agile and more specifically test driven development. In preparation I’ve been exchanging emails with one of their leads on what he is hoping they will get out of this time. He’s a great developer who has recently been exposed to agile practices through his collaboration with Carbon Five. I think his comments are insightful and widely relevant so with his permission I am sharing them with you here.
<snip>
[What we want to accomplish is:] 1. Elucidate what it means to be agile, how it changes the way we think about problems, and show how that thinking is actually practical
…
* WRT #1, a very important and powerful thing I’ve learned from the last few months is to think about the measure of progress as user/customer “visible” artifacts. This really flies in the face of how many of us traditionally think about “progress”; typically progress is this piece of infrastructure or that feature that is required for some future thingy. Concentrating on “user”-visible artifacts gives concrete reward, ability to adapt to problems early, and doesn’t call into existence things that aren’t strictly necessary. It also requires trust in your ability to refactor (ruby is great at that!) and solve problems in the future (we’re all reasonably smart).
Typical objections include:
- “but I’m working on something that isn’t user-visible” — we should address the notion of “user” to be stakeholder, developer, etc. and “feature” to be a concrete artifact
- “but we need all these pieces of infrastructure are required for user-visible feature X!” — you can still decompose the feature into smaller, if not completely useful, pieces. Some of these pieces imply various pieces of infrastructure. By building your infrastructure gradually you always have something you can use *now*, and reduce the risk of ended up with something that’s not quite right.
- “but then where do I put things like design discussion, generating a test plan, and so on?” — some ahead of time design dicussion is useful and necessary, but specifics design decisions are *part* of the implementation of the feature. Testing is *not* a separate step; it’s the definition of what it means for something to be complete. Agile tends to wrap all these things into discrete micro-waterfall units.
- “but I’m building this experimental thing that doesn’t directly contribute to ‘user’ experience” — perhaps there’s another way to look at it, and perhaps it might make sense to think about what specific problem you’re trying to solve. If you can state the problem, you can state a story that implies that it is solved. If not, what are you really doing?
* “Embrace change” is also *really* important. In an environment like ours it’s not useful to get upset if requirements change. Agile methodologies have changing requirements as their primary assumption. Hence the “useful right now” and evolutionary process. [Interesting to note that we built the automated [system] like this, one story at a time, and came in _ahead_ of schedule]
* “Trust yourself” is important too; assume you can refactor and adjust later if you made the wrong decision. This is hard to do, but easier with practice.
* “Communication and collaboration” — In person is way more efficient, and these methodologies tend to favor it. If at an impasse, just start and see where you end up. That’s what we did. A common concern here is that it might be hard to coordinate several teams into a single release, but solutions exist.
* WRT agile practices, something I’ve learned to do in the last few months that is valuable is timeboxing. Totally underrated!
One further thought:
* We can’t afford to *not* take an agile approach (rapid small value-adding changes towards a goal) because 90% of the time when you’re a startup like us you probably won’t end up with (or even want to end up with) what you originally thought you wanted.
</snip>
Java has been the tool of choice for enterprise application development for many organizations for over 10 years. We are seeing more and more applications that are aging and suffering for it. We have worked with several recent clients to breathe new life in to these legacy applications so they can evolve and grow with the business they support.
This is the first of a series of articles about improving the quality of existing software and the processes that produce it.
Why Not Rewrite
We’ve all encountered software that we would rather just rewrite than try to fix. The problem may be poor design, poor implementation, lack of tests, bugs or any of a host of other software ills. We see ahead of us the pain of working with a poor quality system and the risks to schedule and quality of end product it creates.
At Carbon Five, we have participated in many successful projects to rewrite applications for our clients. Sometimes starting with a clean slate and all the lessons learned from a previous effort is exactly the right approach. Often it is not.
In our experience the greatest obstacle to a rewrite effort is that the existing system is the only accurate record of the requirements for the system. Applications evolve over years to meet the changing needs of a organization and its users. Many small decisions are made and captured in the functioning application only. Rewriting to reproduce the features of an existing system can be a very difficult effort to define and scope.
Usually the troubled application will continue to be use while a rewrite effort is underway. Often support, maintenance and feature development will need to continue on that application even while a rewrite is under way. A rewrite will compete for development resources and will be chasing new feature development on the existing application.
Not everything in the troubled application is rotten. There are often good pieces, or at least components that are reliable and well understood. It would be great to not have to rewrite them.
How can you be sure that a rewritten application will be so much better than the one it is replacing? All too often, the source of poor software quality is poor process and practices. Unless you fix those problems it is not worth embarking on a new software development effort.
How to Rescue
Sometimes a decision to rewrite is made because rescuing an application seems too daunting. Where do you start? How long will it take? Here are some high level thoughts from our experiences with our customers. We will get in to more detail on specifics in later articles.
Set Short Term Goals
Be sure to set demonstrable short term goals with meaningful names. When your job is to make the system “better” your job is never done. You need discrete goals with names that you can finish, demonstrate, release and feel good about before moving on. Easily communicated goals help to show management that the effort is progressing successfully and help your team build confidence in an often daunting task.
On one of our projects we set a goal of having a unit test in place for a component in our service layer. “Service Test” would be a fine name for that goal. In a system with no testing support, this was a big task that once completed opened the door for much easier testing.
Don’t Let “Refactor” Become a Dirty Word
A lot of the work you will do when rescuing an application can be characterized as refactoring:
Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.
Martin Fowler, http://www.refactoring.com
Don’t let “refactor” become a dirty word. It’s too easy to say, “Yesterday I worked on refactoring our persistence layer. Today I’m going to keep working on that.” Again, come up with a name or description for the work you are doing that enables others to understand what you are doing and for you to describe progress through that task. Refactoring is one activity of many that you will engage in during this process. Avoid using “refactor” in the names of your short term goals.
Introduce Testing
A common attribute of many struggling applications is conspicuous lack of automated testing and the design and infrastructure to support it. We have found that the many changes required to support and write automated unit and integration tests are exactly the changes an application needs to get healthy. They include:
- Automating and streamlining build systems
- Decoupling application components
- Minimizing dependencies on application server features
- Building test data sets
- Continuous integration
- Building a team interest in quality
Each of these items is challenging to implement. You can’t do them all at once yet they are dependent on each other. Fortunately there are good open tools and practices available that provide a helpful bootstrap including Maven for convention-based builds, EasyMock to test code with lots of dependencies, Spring for component management outside of an application server, and CruiseControl for continuous integration (not to ignore the excellent almost-free options like TeamCity and Bamboo).
Introducing testing to a legacy application with no support for it is a daunting challenge and slow going at first. We’ve had good success in the past by first cleaning up the build and adding the ability to run tests, then finding a place in the application to break apart components and wedge a first test in. With a running test, you have something to automate so you can get continuous integration running. You also have a lot more insight into the architectural changes that will both improve the health of the application and make it easier to add more tests.
We have a lot more to say about this topic and specific experiences to share in coming articles.
A Culture of Quality
Another common attribute of a struggling application is that the team maintaining and developing the application is in continuous firefighting mode. They’ve lost hope of making fundamental improvements to their system and on a daily basis are fixing emergency bugs and hacking in features with their fingers crossed that they are not breaking something else.
Teams working in this mode collaborate poorly with product management. Product management feels that the developers are resistant to new features. Getting releases out is painful and unpredictable because defect rates are high.
These teams also collaborate poorly with each other. Developers tend to become isolated from each other as sole maintainers of one application component or another. They do not talk about how to make things better and become resigned to living with the status quo.
To be successful in a rescue mission or even a rewrite, you have to turn the firefighting culture into one where developers value quality and work for it daily. They should be excited to make things better and be engaging each other with ideas and practices to get there.
Again, this can be a very difficult effort. Sometimes it requires dramatic measures. In our experience this includes:
- Changing a workspace to remove barriers to casual conversation
- Relocating developers to the same physical location
- Hiring new blood and firing those resistant to change
- Pair programming
- Book clubs and study groups
These changes are often the most painful to make. As a consulting company Carbon Five can advocate for these changes, teach pair programming and run a study group, but the real changes have to come from within. We have seen the most success in making these changes when there is an commitment to improve from the business and champions of this effort in management.
Alan Cooper spoke last night around the corner at Hot Studio at an event organized by the San Francisco chapter of IxDA, the Interaction Design Association. His talk was titled “An Insurgence of Quality” and was similar to the talk he gave at an IxDA event in Savannah in February.
Cooper talked for a while about the concept of craftsmanship as it relates to designers and developers. He emphasized that craftsmanship prioritizes quality over cost and time. While keeping costs down and accomplishing tasks in a timely matter are important, craftspeople do not sacrifice quality for either. They take the time and spend the money that it takes to do it right.
The Triad
He proposes that interaction designers and programmers are craftspeople who should unite around their common commitment to quality and demand that business take the time and spend the money to do things right. Doing things right in his proposal means a three-stage process of:
Interaction Design » Design Engineering » Production Engineering
Interaction Design starts with interaction designers talking to stakeholders, both customers and users, to understand the domain and behavioral requirements for software. They then design behaviors for the software based on this primary research.
Design Engineering starts with programmers working with interaction designers to evaluate the feasibility of the designed behavior, iterating over the designs in collaboration with the interaction designers. In this phase, programmers also hash out how the final system will work through a series of short iterations of development focused on identifying and solving technical problems. This product of this work is not production code, it is the definition of how the final system should be built. He feels that Agile methods are very appropriate for this phase.
Production Engineering is the process of cranking out the final product based on detailed written specifications produced in the prior two phases. Cooper claims that since all the details have been worked out, both in behavior and technology, this phase will be maximally efficient — a straight line of productive software development from start to delivery. He feels that a process like RUP is appropriate for this phase.
Cooper calls this process the Triad and elaborates on his web site.
His split between Design Engineering and Production Engineering is based partly on his claim that programmers can fundamentally be divided into two types — those who above all else want to do it right and those who want to get it done and out the door. The ones who want to do it right are the craftsmen and are best suited to be Design Engineering practitioners. The ones who want to get it done are best suited to be Production Engineering practitioners.
Hmmm
Cooper seems to have come a long way in his understanding and appreciation of Agile practices since his notorious 2002 conversation with Kent Beck. Iteration, collaboration and feedback are all important characteristics of his Design Engineering phase.
However, it’s not clear why Production Engineering would not benefit from an Agile process focused on frequent delivery of working software and tracking toward an end goal that has good definition. It’s also not clear why Design Engineering shouldn’t produce early versions of working software that can be tested with users and be the foundation of the final delivered software.
Some projects may benefit from his phasing. I could see his process benefiting a year long initiative to build shrink wrapped software where there is a desire to offshore a portion of the development.
But web software development does not work like that. You can release web software early and often without compromising quality. Yes, heed Cooper’s warning not to rush prematurely to market with ill-conceived products. Stay focused on what matters to users and let them use it as early as you can. Cooper himself cites Sergey Brin of Google as saying, paraphrased, “that early on they were in no rush to have people try Google today. Tomorrow it would be better. Tomorrow would be fine.” Google was in fact out early with something simple but well-considered in users’ hands. They’ve evolved their product daily and weekly to what it is today. They are proof that releasing early does not necessarily mean prematurely.
Cooper’s attempt to place interaction designers between programmers and their customers and users is also troubling. He claims that programmers don’t want to talk to users which we know is simply wrong. In our experience, the further we are from our customer and end user in a project the greater the likelihood the project will go awry. I’d rather see an approach that figures out how to get programmers and interaction designers understanding the needs of their customers and users together and using that understanding to direct work in their specialized areas of expertise.
As a programmer, Cooper can be a bit hard to swallow. He definitely encourages an “us and them” perspective with his tongue-in-cheek comments about the inscrutability of programmers and their work. Of course he does the same for other relationships with a slew of disparaging claims about management and business executives.
Yet it’s encouraging to hear him emphasize the importance of true collaboration between designers and programmers through the life of a project. I hope his understanding of programmers and their craft continues to evolve to support this goal.
I’ve been enjoying using has_finder by Nick at Pivotal Labs and recently added pagination to the application I am providing volunteer time to for the San Francisco Bike Kitchen.
I like the semantics of paginating_find over will_paginate but am not married to that choice if anyone wants to try to convince me otherwise.
In any case, I wanted to paginate the finders created by has_finder. I wrote a simple ActiveRecord mixin that provides a class method ‘acts_as_paginated and is used in your model like:
class Visit < ActiveRecord::Base belongs_to :person acts_as_paginated :size=> 10 has_finder :for_person, lambda { |person| { :conditions => { :person_id => person}, :order => 'datetime DESC' } } has_finder :in_date_range, lambda { |from,to| { :conditions => [ "visits.datetime >= ? and visits.datetime <= ?", from, to ] } } end
Client code looks like:
from, to = Date.new(2008,2,1), Date.new(2008,2,3) visits = Visit.for_person(@person).in_date_range(from, to).paginate(:page => 3)
The mixin looks like:
# Provides Model.paginate(options={}) to allow # the paginating_find plugin to be used with the has_finder plugin. # # Usage: # acts_as_paginated # acts_as_paginated :page => 2, size => 10 # # Install: # Save this file as lib/acts_as_paginated.rb and require 'acts_as_paginated' # in environment.rb. # module HasFinder module PaginatingFind #:nodoc: def self.included(mod) mod.extend(ClassMethods) end module ClassMethods def acts_as_paginated(options={ :page => 1, :size => 20}) cattr_accessor :paginate_defaults self.paginate_defaults = options self.class_eval do extend HasFinder::PaginatingFind::SingletonMethods end end end module SingletonMethods def paginate(args={}) options = self.paginate_defaults.clone options[:page] = args[:page].to_i if args[:page] options[:size] = args[:size].to_i if args[:size] find(:all, :page => { :size => options[:size], :current => options[:page], :first => 1 }) end end end end ActiveRecord::Base.send(:include, HasFinder::PaginatingFind)
It also works to paginate associations:
@person.visits.paginate(:size => 4)
We’ve been enjoying using Markaby on our recent rails projects. When bootstrapping with scaffolding there’s a common recipe of turning ERB views into Markaby and then refactoring views with Markaby helper functions.
One project wrote a rake task that converts ERB views to Markaby. Then Ingar created a plugin that re-implements the base Rails scaffolding to create Markaby views that output the same HTML as the base Rails scaffolding. I took his work one standard refactor further to introduce a MarkabyHelper with helper functions for the layout of form inputs and model views.
Usage
./script/plugin install http://svn.carbonfive.com/public/carbonfive/rails/plugins/markaby_scaffold/trunk/
Use it as you would regular Rails scaffolding:
./script/generate markaby_scaffold Post title:string body:text published:boolean
Prerequisites
markaby_scaffold requires that you install Markaby as a Rails plugin.
Gravy
Unfortunately, Markaby does not appear to be getting the love that it deserves. There is an issue using Markaby with Rails 2.0.2 for which Randy has submitted a patch. This plugin includes a monkey patch for Markaby and Rails 2.0.2 that provides this fix so you don’t have to apply it yourself.
Additionally, the generated MarkabyHelper includes support for using Markaby in your helpers, provides a quick start for refactoring the scaffolding layouts and includes test cases to get you started testing your Markaby helpers.
We’ve kicked off a new project that extends well beyond standard webapp frameworks and tools including significant instant messaging integration. Portions of the project will use Ruby on Rails for web functionality. I know that the open source Jabber software from Jive Software is excellent quality and feature rich. Would JRuby be a good tool for leveraging these Java libraries in a Ruby environment?
Here’s a toy chat client and parrot bot I wrote for the Google Talk service using JRuby and the Java Smack API from Jive’s open source portal ignite realtime.
I’m releasing a first version of JRubyGems. This post is documentation until I come up with something better.
JRubyGems allows you to package RubyGems on your Java application classpath. It removes the file system dependencies on locally installed gems that get awkward when using JRuby and dependent gems in a Java environment.
Use JRubyGems as a replacement for RubyGems. Instead of
require 'rubygems' gem 'activerecord'
use
require 'jrubygems' gem 'activerecord'
Behind the scenes JRubyGems requires RubyGems and injects behavior specific to finding gems on the Java classpath if they are not already installed.
You make JRubyGems available to your application by putting the jar on your application classpath.
Source is available from Subversion at https://svn.carbonfive.com/public/carbonfive/jruby/jrubygems/trunk/. The tests in ruby/test/test_jrubygems.rb illustrate JRubyGems’ behavior.
The jar at http://mvn.carbonfive.com/public/com/carbonfive/jruby/jrubygems/0.3/jrubygems-0.3.jar is all you need to try it out in your own scripts. With JRuby you can require jar files that are in your load path to get them on your classpath and make Ruby files in the jar available on your load path. So if I have a folder lib/ in my load path with the file lib/jrubygems-0.3.jar, I can use JRubyGems directly from a JRuby script with:
require 'jrubygems-0.3.jar' require 'jrubygems'
Most Java applications using embedded Ruby will have their own classpath management strategy. Just make sure the JRubyGems jar gets on the classpath of your application. If you are using Maven, you can use JRubyGems from the Carbon Five repository:
<repositories> <repository> <id>c5-public-repository</id> <name>Carbon Five Public Repository</name> <url>http://mvn.carbonfive.com/public</url> </repository> </repositories> <dependencies> <dependency> <groupId>com.carbonfive.jruby</groupId> <artifactId>jrubygems</artifactId> <version>0.3</version> </dependency> </dependencies>
If you are using Maven you can also require your JRuby dependency with:
<dependency> <groupId>org.jruby</groupId> <artifactId>jruby-complete</artifactId> <version>1.1RC1</version> </dependency>
JRubyGems expects a gem to be available on your application classpath as a .gem file (the distribution package for a gem) under the classpath location /gems. This means you can have a gems/ folder on your classpath with a bunch of .gem files in it, a jar with all your .gem files under /gems or a jar per .gem file with each file under /gems in each jar. (A limitation in the JDK’s ClassLoader.getResources method which only returns file system locations for a passed-in empty String prevented me from allowing .gem files in jar roots.)
For our migration application, I built a jar for each gem dependency and deployed it to our central Maven repository. My Maven build references these dependencies in my project POM just as I would a jar dependency.
<dependency> <groupId>com.carbonfive.jruby.gems</groupId> <artifactId>activerecord</artifactId> <version>1.15.6</version> </dependency>
I just grabbed the .gem files from my local JRuby installation’s gem cache dir at jruby-1.1b1/lib/ruby/gems/1.8/cache/ to build these jars for now.
I like that Raven takes exactly the opposite approach - managing jar dependencies as gems.
Christian and I have a side project at Carbon Five to create a standard mechanism for managing database migrations for our Java projects. The first implementation defines a migration as a SQL script to be run against the target database. I thought it would be cool to support ActiveRecord migrations too, especially for providing an easy way to do the data migrations that accompany a schema change.
My prototype of this idea went well (more on that some other time) but I quickly ran into an issue with the gem dependency for ‘activerecord-jdbc-adapter’ and its dependencies ‘activesupport’ and ‘activerecord’. The issue is that my Java project was nicely portable but the gems create a dependency on the runtime system having specific gems installed. If I install the gems locally, I also have to install them on other runtime systems like our continuous integration server.
I found another example of a JRuby user running into this problem when reading about this example of using the RedCloth Ruby library to format text in a Spring/Java application. Note the “ugly underbelly” footnote.
In the Ruby world, this is the way things work. Large Ruby applications (especially Rails applications) often rely on system services like cron to do their work. These applications have their own mechanisms for setting up new runtime systems. In the Java world, we expect to package our application with all of its dependencies and deploy it to little more than a JVM and webapp container.
Java has the classpath abstraction to address minimizing file system dependencies while Ruby uses the file system directly and extensively. I’ve seen a couple approaches for addressing this mismatch between Java and Ruby. All of them involve taking some set of resources packaged in a Java archive (jar, war), extracting it to a file system location at runtime and configuring Ruby to use the resources at that location. The “jruby-complete” JRuby distribution unpacks the core Ruby libraries into ~/.jruby/. GoldSpike, the Rails plugin for packaging Rails applications as war files, bundles gems in WEB-INF and configures GEM_HOME at runtime to use the gems from the unpacked war.
I decided to take a similar approach with JRubyGems - bundle gem dependencies in the application classpath and install them on the local file system on demand if they are not already available. As I expect happens with many JRuby projects, it took a little Java and a little Ruby and works quite well.
The main flow for RubyGems to load a gem that has not yet been loaded for an application is:
- Kernel::gem
- Kernel::activate_gem_with_options
- Gem.activate
- Gem.source_index.find_name
- Gem.activate each dependency of this gem first
- Finish activating this gem
JRubyGems replaces the Gem.activate implementation to insert a couple steps that:
- Check if the gem is locally installed
- Search the Java classpath for the gem source if not installed
- Install the gem locally
- Continue with the base Gem.activate implementation
This ensures that gems and their transitive dependencies are installed and loaded.
The classpath searching is implemented in Java.
Release History
0.3 2008-01-28
- Upgrade to use JRuby 1.1RC1 and RubyGems 1.0.1 (bundled with 1.1RC1)
- Added test for gem that bundles a jar in lib/ (hpricot)
0.2 2008-01-07
- Resolved issues with transitive dependency installation
- Force install of classpath gems to avoid dependency errors
- Additional logging of install locations
0.1 2008-01-04
Initial release.
