Archive for the 'Software Development' Category

Cucumber, what’s the point?

Friday, June 22nd, 2012

Last tuesday, FedEx dropped off my latest Amazon impulse buy. It is a book called The Cucumber Book: Behaviour-Driven Development for Testers and Developers. This is one of the recent books published by the Pragmatic Programmers. It covers a popular Ruby based testing framework called Cucumber (.) Now cucumber makes a few bold claims on their site. They claim that tests can be written in plain english, and actually test real code. They also claim that with minimal training, even a non-QA person, like a product manager, could write these tests, which would double as specifications.
Read the rest of this entry »

Zembly makes social apps simple

Thursday, March 19th, 2009

Widgets have definitely become “the next new thing”. The small snippets of functionality can be plastered just about anywhere on the web, your homepage, facebook profile, blog, etc. Widgets tend to range from completely frivolous decorations to dead useful mini-apps to games.. the list goes on.

As widgets have gained in popularity, various companies have created tools to assist in their creation. Google has created a tool that allows you to either create a widget from an existing template or create your own from scratch. The template based widgets don’t even require any programming knowledge. Just fill out the form and presto. Yahoo has a similar offering. But the most interesting so far has to be an offering from Sun called zembly.

Zembly provides an all in one widget creation solution. You can choose what kind of widget you want ot build. You can choose from a normal web widget, Facebook application, OpenSocial application, or even a Meebo application. They then provide a wizard to walk you through configuring your application. They provide an interface to most of the major data services (google maps, google translate, amazon retail search, and many others) as well as a way to add any other public service you may need.

You don’t have to worry about having a server to host the widget or application on. Zembly provides free hosting with any account. Not only that, but they do allow you to fetch the source code for your application should you choose to host it elsewhere.

One last interesting thing that zembly offers. Anybody can see, and contribute to anybody’s widget. Think of this as socially generated software. Anybody can collaborate on any widget or app. They can use any app as a starting point for their idea and so forth. Zembly does provide a way to disable this, and keep your app private if you so choose, but it appears that this feature may cost in the future.

So there you go. Zembly provides the tools, the hosting, and the collaboration to ride the widget wave as far as it can go. They have the best tools I have seen. They have fantastic integration with facebook. They provide hosting for those who need it, and source code for those who don’t. Now go build your widget!

-Joe

Prospering with ruby vs. haskell

Wednesday, March 26th, 2008

As previously mentioned, I am learning haskell. In that endeavor, I am trying to cross the chasm from “tutorial following” to actual real projects (albeit, very small projects). My latest project is a simple simulator for my prosper.com account. For those who don’t know, prosper.com allows people to make smallish loans to each other with terms of three year repayment. Money amounts range from $50 to $25,000, and interest rates are negotiated in an auction. As a lender, I want to know what return on investment I am likely to receive given various scenarios.

Now on to the show

My first stab at the simulator was done in ruby. This gives me a working model, and the ability to compare and contrast some of the design requirements that functional programming, and specifically haskell will impose.

Ruby

First I needed a function to generate random rates, simulating the auction style rate negotiation.
def get_new_rate
   return MIN_RATE + rand(RATE_WINDOW)
end

where MIN_RATE is defined as the minimum rate I am willing to lend at (8.0%), and RATE_WINDOW is defined as the spread between my minimum rate, and the highest rate I am interested in lending at (20.0%).

Second off, I needed a function to generate a number of loans given a certain account balance.
def add_loans(loans, account_balance)
  new_loan_count = account_balance / INITIAL_PRINCIPLE
  new_loan_count.to_i.times do
    rate = get_new_rate
    loans << {:principle => INITIAL_PRINCIPLE, :rate => rate, :min_payment => calc_minimum_payment(INITIAL_PRINCIPLE,rate)}
    account_balance -= INITIAL_PRINCIPLE
  end
  account_balance
end

where INITIAL_PRINCIPLE is set to the amount that I am willing to lend ($50) in each loan. (Read this for an explanation of why I only lend $50.)
This function calculates how many loans I can generate from the given account balance, then creates each one. The new loans are appended to the collection of loans that was passed in as an argument. The calc_minimum_payment function simply determines what the minimum payment will be each month.

I then needed a function that would calculate the payment on the loan – particularly at the end of the loan when the payment may be less than the minimum payment.
def calc_payment(loan, months=1)
  if loan[:principle] < loan[:min_payment]
    payment = loan[:principle]
  loan[:principle] = 0
  return payment
  end
  
  interest = loan[:principle] * loan[:rate] / 100.0 / 12
  loan[:principle] -= loan[:min_payment] - interest
  return loan[:min_payment]
end

Given these functions, I can now create the simulation
account_balance = ARGV[0].to_f if ARGV[0]
account_balance ||= 0.0
  
monthly_deposit = ARGV[1].to_f if ARGV[1]
monthly_deposit ||= 100.0
  
number_of_years = ARGV[2].to_i if ARGV[2]
number_of_years ||= 1

First grab the scenario parameters from the cmdline. monthly_deposit is how much money to add to the account balance each month (in addition to the payments from the outstanding loans)

loans = []
for i in (1..number_of_years*12)
  old_loans = loans.size
  account_balance = add_loans loans, account_balance
  print "Month #{i}\n"
  print "Number of loans: #{loans.size} (#{loans.size - old_loans})\n"
  print "Average Rate: #{calc_average(loans.collect {|i| i[:rate]})}\n"
  income= loans.inject(0) {|bal, i| bal + calc_payment(i)}
  print "Account balance: #{account_balance}\n"
  print "Income: #{income}\n"
  account_balance += income + monthly_deposit
  print "Account value: #{loans.collect {|i| i[:principle]}.inject(account_balance) {|sum, i| sum + i}}\n"
  print "\n"
  loans.delete_if {|item| item[:principle] == 0}
end

Then run the simulation, and print out various statistics for each month of the simulation.

So that’s the simulator in ruby. It is not “perfectly optimized” for ruby, because I wanted to keep it somewhat close to the structure that I would use for haskell. See the link below for full source.

Haskell

I tried to keep the architecture of the haskell version as close to the ruby approach as was possible. As a consequence, many haskell people may look at this and balk. My apologies in advance.
First I needed some constants and a struct to keep the relevant loan data in
minRate = 8.0
maxRate = 20.0
initialPrinciple = 50.0
periods = 36
  
data Loan = Loan {principle :: Double, rate :: Double, minPayment :: Double}

Then comes the function used to simulate the rate auctions
-- Generate a random rate within the "rate window"
getNewRate :: IO Double
getNewRate = do randomRIO (minRate, maxRate)

Where I calculate some random number between the minRate and maxRate. Note the type – IO Double. For all you non-haskellites, that means that the function will be using a monad inside of itself. In this case, the monad is randomRIO. The monad allows you to call randomRIO multiple times, and get different numbers each time. Useful that!

Then I have the loan creation functions

-- figure out what the minimum payment will be on a given loan
calcMinimumPayment :: Double -> Double -> Double
calcMinimumPayment p i = (r * p *(1+r)^periods) / ((1+r)^periods - 1)
                         where r = i / 12.0 / 100
 
-- create a new loan
newLoan :: Double -> IO Loan
newLoan p = do
          i <- getNewRate
          let m = calcMinimumPayment p i
          return (Loan p i m)

As in the ruby version, create a new loan, then populate the structure with the rate and minimum payment. Note the type for calcMinimumPayment doesn’t specify IO… that means this is a “clean function” and can be called anywhere. newLoan however is a monad function – because it calls getNewRate. Since newLoan uses a monad function, it has to return a monad itself.

Here’s where things had to deviate from how I did them in ruby. Since haskell has immutable values, I couldn’t modify the loans. I had to create new loans, and collect them into a new structure. Here is where the new loan is created, given the state of the provided loan.
-- Given a loan, make a payment and create a new loan with the remaining principle
calcPayment :: Loan -> Loan
calcPayment l = if principle l > minPayment l
      then Loan (principle l - p) (rate l) (minPayment l)
      else Loan 0 (rate l) (principle l) -- mark this as the last payment
      where
        i = (principle l) * (rate l / 100 / 12)
        p = minPayment l - i

Again, here’s a “pure function”. It can be called anywhere, and any time.

Now, given an account balance, create as many loans as I can, and return them as a collection of Loans.
-- Take the current account balance, and make as many loans as possible from it
makeLoans :: Double -> IO [Loan]
makeLoans bal = if bal >= initialPrinciple
      then do
        l <- newLoan initialPrinciple
        ls <- makeLoans (bal - initialPrinciple)
        return ([l] ++ ls)
      else
        return []

Note the recursive call to continue building the list. I am finding that functional programming relies on recursion a lot more than OOP.

This is another portion of code where I had to deviate. Here is where I actually parse the passed in loans, and return a new array of updated loans, and a new account balance. This is probably the most un-haskellish function of the group, and definitely needs some work.
-- make payments on the given loans, and return the updated loans, and resulting total payments
collectPayments :: [Loan] -> ([Loan], Double)
collectPayments loans = (filteredLoans, payments)
      where
        clearStaleLoans = filter (\x -> minPayment x > 0) -- remove any loans that have been fully paid back
        filteredLoans = clearStaleLoans (map calcPayment loans= sum (map minPayment filteredLoans)

Then a function that runs through each iteration of the simulation – i.e. each month. This has to be its own function so that it can recursively call itself to continue the simulation.
-- run through a loan scenario, reinvesting returns for 'term' months. Print out various statistics on the account
run :: Double -> [Loan] -> Int -> Double -> IO Double
run startingBalance loans term monthlyDeposit = if term <= 0
        then return startingBalance
      else do
        l <- makeLoans startingBalance
                              let (newLoans, newPayments) = collectPayments (loans ++ l)
        let newPrinciple = (initialPrinciple * fromIntegral (length l))
        let newBalance = (startingBalance - newPrinciple + newPayments)
        let loanValue = sum (map principle newLoans)
        let averageRate = (sum (map rate newLoans)) / fromIntegral (length newLoans)
        putStr $ unlines ["Term: " ++ show term, "Loan count: " ++ show (length newLoans), "Average Rate: " ++ show averageRate, "Loan Value: " ++ show loanValue, "New balance: " ++ show newBalance, "New Principle: " ++ show newPrinciple, "New Payments: " ++ show newPayments,"---------"]
        bal <- (run (newBalance + monthlyDeposit) newLoans (term-1) monthlyDeposit)
        return bal

Note how, although run is a monad function, a majority of its processing is non-monadic. In theory each of those ‘let’ statements could run in parallel.

Finally a “main” function to get the works rolling
main :: IO ()
main = do
        args <- getArgs
        let accountBalance = if(length args > 0) then read (args !! 0) :: Double else 300.0
        let monthlyDeposit = if(length args > 1) then read (args !! 1) :: Double else 100.0
        let term = if(length args > 2) then read (args !! 2) :: Int else 1
        putStr $ unlines ["Starting balance: " ++ show accountBalance, "Starting run", "----------"]
        endingBalance <- run accountBalance [] (term * 12) monthlyDeposit
        putStr $ unlines ["Ending Balance: " ++ show endingBalance]

Summing it up

Well, comparing and contrasting these two scripts is giving me a new appreciation for both languages. Each script could be refined to better match its underlying language, but the goal was to keep the code as close as possible to maximize comparability. If enough people ask, perhaps I’ll refine each script.
Hopefully some comparison of the two scripts will help another budding haskell developer wrap their head around this powerful, but oh so different language.

Here is the full ruby source code – prosper.rb (right click – “Save As”)
Here is the full haskell source code – prosper.hs (right click – “Save As”)

-Joe

Functional Invasion

Sunday, December 2nd, 2007

If you write software, or even pretend to write software, you have probably heard of functional programming. You may even know what functional programming is. If you do, then bear with me for a moment while I summarize things for those that don’t.

Languages tend to fall into one of three different paradigms. Procedural (c, perl, php, javascript), Object Oriented (c++, java, ruby), and Functional (haskell, ml, erlang). Now, this isn’t to say that a language only follows one specific paradigm. One can take java or c++ and implement code in all three paradigms. Same with almost any other language, but there tends to be a general layout to a language that makes it more suitable to one paradigm or another. So what are these paradigms?

Procedural – This is where you are effectively writing, step by step, what you want your application to do. You rarely have any abstraction, and your code tends to be fairly straightforward. Since everything is explicit, it can be optimized, and run very tightly. However, the responsibility is on you as the developer to make it so. You could just as easily (perhaps more easily) make your code brittle, bug ridden, and slow.

Object Oriented – This is where you have your code logically broken up into areas of responsibility. An object is a component of data, accompanied by the various bits of functionality that can act upon it. This paradigm tends to result in isolated functionality that is reused as the data is reused. This will typically lead to more maintainable projects with flexible abstractions allowing for minimal code doing maximal work. On the other hand, abstraction can be computationally expensive, and result in “magical functionality” that is very difficult to trace through. Again, it requires a responsible developer.

Functional – This is where you are concentrating on functions (in a mathematical sense) or algorithms. You build your program as a series of “data pipelines” that will take a given input and provide a desired output. These functions are written such that if the inputs remain the same, any number of calls to the function will always result in the same output. This paradigm, when properly followed, can allow for a compiler to make certain assumptions about the program being compiled, and can lead to some very radical optimizations. It also tends to discourage some of the more evil behaviors of programmers (such as modifying global data, having one function do many unrelated things, etc). The cost is that it requires a very different mindset when writing your programs. It can also be very tedious to write functionally styled code in a “non-functional” language.

The interesting thing about Procedural vs. Object Oriented paradigms is that they are still related. Inside the functions of an object, procedural code is used to do work. All you are really doing is introducing some forms of organization to the code layout. When you look at Procedural vs. Functional paradigms, you will find they are fundamentally different. Procedural code will lay out, step by step, what data you are using, and what you want to do to that data. In a functional language, this is not really the case. Instead, you will lay out what you want to do, and build compositions of functions to tie into each other. Then you will add inputs to the head of the function. Another way of thinking of this would be like using the pipe (‘|’) in the *nix Bash shell. Given data from some program, you want to transform it using other programs (like grep, awk, or sed) and then capture the results. Each of those utilities are like a function. Given the same inputs, they will always generate the same output. Piping the outputs of one function to the inputs of another is more or less the premise of functional programming.

So, if functional programming is so different than what everyone is used to, why does it matter? Well, it turns out that functions written in this way are typically easier to test. Since each function has no side effects, unit testing can be used effectively. Once a set of functions are well tested, then large majority of errors will be in the composition of functions, and these are easy for a compiler to catch in any relatively type safe language. Reuse can be maximized, resulting in a smaller, better tested, more maintainable code base. If you are using a functional language, then the compiler can do some very interesting things for optimization, and parallelization. Even if you are not using a functional language, many benefits can be had. The STL uses functional programming techniques extensively to allow c++ developers to enjoy some of the benefits. Still, to maximize the enjoyment of functional programming, one must use a functional language. Only then will the compiler leverage the assumptions that can be made about your code. It can, for example see that you always call a certain function with the same parameters every time, and replace that function call with the result directly. It can use cues that you provide to detect that a certain function can be run in parallel and automatically tie a thread-pool to the function.

So if you have looked at any of the functional languages and scoffed at the learning curve. Perhaps consider taking a second look. In fact, sometimes you will find the learning curve to be pretty reasonable. I recently ran across the book Programming Erlang: Software for a Concurrent Worldamazon.com. Once again the Pragmatic Programmers have worked to produce a book that makes learning fun and interesting. The learning curve is manageable, and the examples in the book are more than just trivial exercises. Erlang is an interesting language that makes working in a concurrent environment trivially simple, and on todays multi-core processors, that is becoming important. It can also pave the way to understanding more involved languages like Haskell. It can perhaps even allow you to understand the techniques of functional programming better so you can employ them in your non-functional language more effectively. After all, isn’t being more effective what this is all about?

-Joe