It seems like every day brings a new round of layoff announcements. Companies, large and small, are having to cut back, save cash, and stop growing. In this environment, any new hires need to be as effective as possible. It’s all about the bang for the buck. This is a time where generalists may have an advantage – wearing many hats makes you more cost effective.

Given this, I’m seeing several job postings that are asking for an awful lot. It is to be expected, and makes sense from the companies point of view. There are, however, times where this “hire overloading” borders on the absurd. Take the following:

*Senior Linux/Unix Developer/Test Engineer*
*Summary:*
Installs and configures clusters of Linux based application and database servers. Drafts and executes test plans of Linux related software on clusters. Experience with Linux application cluster design, administration and tuning (including san) required. Experience with virtualization technologies required.
*Details:*
Strong software development skills in multi-tiered and distributed environments using iterative development process, including 5+ years of advanced programming experience

* Application performance testing plan drafting and execution.
* Experience with usage and customization of open source application performance test tools.
* Multiple Programming Languages: C, C++, Perl, Python, Cold Fusion, JAVA
* 4 year technical degree or higher at an accredited institution.
* Linux Cluster and cluster storage design, configuration and tuning.
* Linux Kernel customization and compilation
* Databases: Mysql, Postgresql, Oracle
* Multiple Operating Systems: Linux, FreeBSD, Windows, etc.
* Experience with multiple virtualization technologies: Xen, VMWare, KVM,
* Excellent analytical and problem solving skills; with the ability to analyze business processes and create application models utilizing project-management standards
* Strong verbal and written communication skills and ability to work effectively both independently and as part of a cross functional team.
* Telecom and/or internet domain knowledge and experience with solid experience with eBusiness processes and/or back-end applications

So, here’s what I read into this request. This company is looking for a QA engineer who has been a mid/high level software engineer, with linux administration experience, fairly advanced enterprise hosting experience, and project management experience. The hiring manager is asking for a chocolate cake that has just the right amount of fish sauce and onions.

This is at least two different career tracks. Any software engineer that has professional experience setting up virtualization clusters and SAN is not likely to be very skilled in one or both areas.

Hiring managers that cram so many diverse requirements into a single job posting really bug me. If you want a generalist, then ask for a generalist. Asking for such specific and wildly different requirements does nothing to increase your chances of finding a quality candidate. It just turns away those quality generalists that are capable of learning your specific system. If you need someone who can set up test environments using a virtualization system, then either train one of your test engineers – if muddling through is acceptable – or hire a sysadmin who has experience with virtualization. If you need someone who can write test harnesses, then hire someone who knows how to write test harnesses and exercise programming interfaces. If the harnesses require an obscure language, then asking for familiarity in that language makes sense. Remember, a good engineer can learn whatever elements are specific to your system. If you want someone who can analyze your current business processes, and help to reorganize them, don’t expect the person to also be down in the code – those are two totally different disciplines. “big picture” people, who can give quality feedback on business process tend to miss the details necessary for writing a good testing suite, and vice versa.

I feel bad for any technical recruiter who sends out job requests like this because though they are just the messenger, they look foolish. The hiring manager looks just plain silly. This was a request that was likely written by a team of engineers. They all looked at what they do, and tried to “fill in the gaps”. Bad practice! Anybody who actually fits that request is either dishonest (resume padding) or probably doesn’t require the depth of knowledge that the team is hoping for. The team, or hiring manager, should request a generalist, who has experience in the main area of the job request. Then during the interview process, explore the candidates willingness and ability to learn. Perhaps mention some of the systems that are required to do the job. Focus on concepts and approaches, not on specific technologies or processes.

I know it’s a tough job market, but come on. A little sanity please.

-Joe

As previously mentioned, I am learning haskell. In that endeavor, I am trying to cross the chasm from “tutorial following” to actual real projects (albeit, very small projects). My latest project is a simple simulator for my prosper.com account. For those who don’t know, prosper.com allows people to make smallish loans to each other with terms of three year repayment. Money amounts range from $50 to $25,000, and interest rates are negotiated in an auction. As a lender, I want to know what return on investment I am likely to receive given various scenarios.

Now on to the show

My first stab at the simulator was done in ruby. This gives me a working model, and the ability to compare and contrast some of the design requirements that functional programming, and specifically haskell will impose.

Ruby

First I needed a function to generate random rates, simulating the auction style rate negotiation.
def get_new_rate
   return MIN_RATE + rand(RATE_WINDOW)
end


where MIN_RATE is defined as the minimum rate I am willing to lend at (8.0%), and RATE_WINDOW is defined as the spread between my minimum rate, and the highest rate I am interested in lending at (20.0%).

Second off, I needed a function to generate a number of loans given a certain account balance.
def add_loans(loans, account_balance)
  new_loan_count = account_balance / INITIAL_PRINCIPLE
  new_loan_count.to_i.times do
    rate = get_new_rate
    loans << {:principle => INITIAL_PRINCIPLE, :rate => rate, :min_payment => calc_minimum_payment(INITIAL_PRINCIPLE,rate)}
    account_balance -= INITIAL_PRINCIPLE
  end
  account_balance
end


where INITIAL_PRINCIPLE is set to the amount that I am willing to lend ($50) in each loan. (Read this for an explanation of why I only lend $50.)
This function calculates how many loans I can generate from the given account balance, then creates each one. The new loans are appended to the collection of loans that was passed in as an argument. The calc_minimum_payment function simply determines what the minimum payment will be each month.

I then needed a function that would calculate the payment on the loan – particularly at the end of the loan when the payment may be less than the minimum payment.
def calc_payment(loan, months=1)
  if loan[:principle] < loan[:min_payment]
    payment = loan[:principle]
  loan[:principle] = 0
  return payment
  end
  
  interest = loan[:principle] * loan[:rate] / 100.0 / 12
  loan[:principle] -= loan[:min_payment] - interest
  return loan[:min_payment]
end


Given these functions, I can now create the simulation
account_balance = ARGV[0].to_f if ARGV[0]
account_balance ||= 0.0
  
monthly_deposit = ARGV[1].to_f if ARGV[1]
monthly_deposit ||= 100.0
  
number_of_years = ARGV[2].to_i if ARGV[2]
number_of_years ||= 1

First grab the scenario parameters from the cmdline. monthly_deposit is how much money to add to the account balance each month (in addition to the payments from the outstanding loans)

loans = []
for i in (1..number_of_years*12)
  old_loans = loans.size
  account_balance = add_loans loans, account_balance
  print "Month #{i}\n"
  print "Number of loans: #{loans.size} (#{loans.size - old_loans})\n"
  print "Average Rate: #{calc_average(loans.collect {|i| i[:rate]})}\n"
  income= loans.inject(0) {|bal, i| bal + calc_payment(i)}
  print "Account balance: #{account_balance}\n"
  print "Income: #{income}\n"
  account_balance += income + monthly_deposit
  print "Account value: #{loans.collect {|i| i[:principle]}.inject(account_balance) {|sum, i| sum + i}}\n"
  print "\n"
  loans.delete_if {|item| item[:principle] == 0}
end

Then run the simulation, and print out various statistics for each month of the simulation.

So that’s the simulator in ruby. It is not “perfectly optimized” for ruby, because I wanted to keep it somewhat close to the structure that I would use for haskell. See the link below for full source.

Haskell

I tried to keep the architecture of the haskell version as close to the ruby approach as was possible. As a consequence, many haskell people may look at this and balk. My apologies in advance.
First I needed some constants and a struct to keep the relevant loan data in
minRate = 8.0
maxRate = 20.0
initialPrinciple = 50.0
periods = 36
  
data Loan = Loan {principle :: Double, rate :: Double, minPayment :: Double}


Then comes the function used to simulate the rate auctions
-- Generate a random rate within the "rate window"
getNewRate :: IO Double
getNewRate = do randomRIO (minRate, maxRate)


Where I calculate some random number between the minRate and maxRate. Note the type – IO Double. For all you non-haskellites, that means that the function will be using a monad inside of itself. In this case, the monad is randomRIO. The monad allows you to call randomRIO multiple times, and get different numbers each time. Useful that!

Then I have the loan creation functions

-- figure out what the minimum payment will be on a given loan
calcMinimumPayment :: Double -> Double -> Double
calcMinimumPayment p i = (r * p *(1+r)^periods) / ((1+r)^periods - 1)
                         where r = i / 12.0 / 100
 
-- create a new loan
newLoan :: Double -> IO Loan
newLoan p = do
          i <- getNewRate
          let m = calcMinimumPayment p i
          return (Loan p i m)

As in the ruby version, create a new loan, then populate the structure with the rate and minimum payment. Note the type for calcMinimumPayment doesn’t specify IO… that means this is a “clean function” and can be called anywhere. newLoan however is a monad function – because it calls getNewRate. Since newLoan uses a monad function, it has to return a monad itself.

Here’s where things had to deviate from how I did them in ruby. Since haskell has immutable values, I couldn’t modify the loans. I had to create new loans, and collect them into a new structure. Here is where the new loan is created, given the state of the provided loan.
-- Given a loan, make a payment and create a new loan with the remaining principle
calcPayment :: Loan -> Loan
calcPayment l = if principle l > minPayment l
      then Loan (principle l - p) (rate l) (minPayment l)
      else Loan 0 (rate l) (principle l) -- mark this as the last payment
      where
        i = (principle l) * (rate l / 100 / 12)
        p = minPayment l - i

Again, here’s a “pure function”. It can be called anywhere, and any time.

Now, given an account balance, create as many loans as I can, and return them as a collection of Loans.
-- Take the current account balance, and make as many loans as possible from it
makeLoans :: Double -> IO [Loan]
makeLoans bal = if bal >= initialPrinciple
      then do
        l <- newLoan initialPrinciple
        ls <- makeLoans (bal - initialPrinciple)
        return ([l] ++ ls)
      else
        return []

Note the recursive call to continue building the list. I am finding that functional programming relies on recursion a lot more than OOP.

This is another portion of code where I had to deviate. Here is where I actually parse the passed in loans, and return a new array of updated loans, and a new account balance. This is probably the most un-haskellish function of the group, and definitely needs some work.
-- make payments on the given loans, and return the updated loans, and resulting total payments
collectPayments :: [Loan] -> ([Loan], Double)
collectPayments loans = (filteredLoans, payments)
      where
        clearStaleLoans = filter (\x -> minPayment x > 0) -- remove any loans that have been fully paid back
        filteredLoans = clearStaleLoans (map calcPayment loans= sum (map minPayment filteredLoans)


Then a function that runs through each iteration of the simulation – i.e. each month. This has to be its own function so that it can recursively call itself to continue the simulation.
-- run through a loan scenario, reinvesting returns for 'term' months. Print out various statistics on the account
run :: Double -> [Loan] -> Int -> Double -> IO Double
run startingBalance loans term monthlyDeposit = if term <= 0
        then return startingBalance
      else do
        l <- makeLoans startingBalance
                              let (newLoans, newPayments) = collectPayments (loans ++ l)
        let newPrinciple = (initialPrinciple * fromIntegral (length l))
        let newBalance = (startingBalance - newPrinciple + newPayments)
        let loanValue = sum (map principle newLoans)
        let averageRate = (sum (map rate newLoans)) / fromIntegral (length newLoans)
        putStr $ unlines ["Term: " ++ show term, "Loan count: " ++ show (length newLoans), "Average Rate: " ++ show averageRate, "Loan Value: " ++ show loanValue, "New balance: " ++ show newBalance, "New Principle: " ++ show newPrinciple, "New Payments: " ++ show newPayments,"---------"]
        bal <- (run (newBalance + monthlyDeposit) newLoans (term-1) monthlyDeposit)
        return bal

Note how, although run is a monad function, a majority of its processing is non-monadic. In theory each of those ‘let’ statements could run in parallel.

Finally a “main” function to get the works rolling
main :: IO ()
main = do
        args <- getArgs
        let accountBalance = if(length args > 0) then read (args !! 0) :: Double else 300.0
        let monthlyDeposit = if(length args > 1) then read (args !! 1) :: Double else 100.0
        let term = if(length args > 2) then read (args !! 2) :: Int else 1
        putStr $ unlines ["Starting balance: " ++ show accountBalance, "Starting run", "----------"]
        endingBalance <- run accountBalance [] (term * 12) monthlyDeposit
        putStr $ unlines ["Ending Balance: " ++ show endingBalance]


Summing it up

Well, comparing and contrasting these two scripts is giving me a new appreciation for both languages. Each script could be refined to better match its underlying language, but the goal was to keep the code as close as possible to maximize comparability. If enough people ask, perhaps I’ll refine each script.
Hopefully some comparison of the two scripts will help another budding haskell developer wrap their head around this powerful, but oh so different language.

Here is the full ruby source code – prosper.rb (right click – “Save As”)
Here is the full haskell source code – prosper.hs (right click – “Save As”)

-Joe

A couple of months ago, when ruby on rails 2.0 came out, I started a project to see if things had really changed all that much. I wanted something that was easy, but useful to me. At work, we had been discussing the pressing need for an easy to use listing of lunch places around our area. So, I decided, not only would I pick up rails 2.0, but I would see how hard it was to do a google maps mashup. Well, after a few days, I had the basics of a site up and running. Then of course, work got busy busy. (Something about launching our new website at work.) Well, in order to start benefiting from at least some of my effort, I spent the last weekend getting everything set up to deploy to my server, so without further ado, I present Lunchspot! Now, I wouldn’t call this a completed effort. In fact, it is only a faint echo of what I had hoped to have going by this time.

So why release it now? Well, because it is still quite useful. You can already submit your favorite lunch spots, comment on them, and see where they are on the map. Soon, I’ll have directions, ratings, sample menus, and “send to a friend” links. But the biggest reason to put it up, and then post about it, is to remind me to work on it, and solicit any help on things like design/layout and graphics (two of my weak areas).

So what is for lunch anyway?

If you write software, or even pretend to write software, you have probably heard of functional programming. You may even know what functional programming is. If you do, then bear with me for a moment while I summarize things for those that don’t.

Languages tend to fall into one of three different paradigms. Procedural (c, perl, php, javascript), Object Oriented (c++, java, ruby), and Functional (haskell, ml, erlang). Now, this isn’t to say that a language only follows one specific paradigm. One can take java or c++ and implement code in all three paradigms. Same with almost any other language, but there tends to be a general layout to a language that makes it more suitable to one paradigm or another. So what are these paradigms?

Procedural – This is where you are effectively writing, step by step, what you want your application to do. You rarely have any abstraction, and your code tends to be fairly straightforward. Since everything is explicit, it can be optimized, and run very tightly. However, the responsibility is on you as the developer to make it so. You could just as easily (perhaps more easily) make your code brittle, bug ridden, and slow.

Object Oriented – This is where you have your code logically broken up into areas of responsibility. An object is a component of data, accompanied by the various bits of functionality that can act upon it. This paradigm tends to result in isolated functionality that is reused as the data is reused. This will typically lead to more maintainable projects with flexible abstractions allowing for minimal code doing maximal work. On the other hand, abstraction can be computationally expensive, and result in “magical functionality” that is very difficult to trace through. Again, it requires a responsible developer.

Functional – This is where you are concentrating on functions (in a mathematical sense) or algorithms. You build your program as a series of “data pipelines” that will take a given input and provide a desired output. These functions are written such that if the inputs remain the same, any number of calls to the function will always result in the same output. This paradigm, when properly followed, can allow for a compiler to make certain assumptions about the program being compiled, and can lead to some very radical optimizations. It also tends to discourage some of the more evil behaviors of programmers (such as modifying global data, having one function do many unrelated things, etc). The cost is that it requires a very different mindset when writing your programs. It can also be very tedious to write functionally styled code in a “non-functional” language.

The interesting thing about Procedural vs. Object Oriented paradigms is that they are still related. Inside the functions of an object, procedural code is used to do work. All you are really doing is introducing some forms of organization to the code layout. When you look at Procedural vs. Functional paradigms, you will find they are fundamentally different. Procedural code will lay out, step by step, what data you are using, and what you want to do to that data. In a functional language, this is not really the case. Instead, you will lay out what you want to do, and build compositions of functions to tie into each other. Then you will add inputs to the head of the function. Another way of thinking of this would be like using the pipe (‘|’) in the *nix Bash shell. Given data from some program, you want to transform it using other programs (like grep, awk, or sed) and then capture the results. Each of those utilities are like a function. Given the same inputs, they will always generate the same output. Piping the outputs of one function to the inputs of another is more or less the premise of functional programming.

So, if functional programming is so different than what everyone is used to, why does it matter? Well, it turns out that functions written in this way are typically easier to test. Since each function has no side effects, unit testing can be used effectively. Once a set of functions are well tested, then large majority of errors will be in the composition of functions, and these are easy for a compiler to catch in any relatively type safe language. Reuse can be maximized, resulting in a smaller, better tested, more maintainable code base. If you are using a functional language, then the compiler can do some very interesting things for optimization, and parallelization. Even if you are not using a functional language, many benefits can be had. The STL uses functional programming techniques extensively to allow c++ developers to enjoy some of the benefits. Still, to maximize the enjoyment of functional programming, one must use a functional language. Only then will the compiler leverage the assumptions that can be made about your code. It can, for example see that you always call a certain function with the same parameters every time, and replace that function call with the result directly. It can use cues that you provide to detect that a certain function can be run in parallel and automatically tie a thread-pool to the function.

So if you have looked at any of the functional languages and scoffed at the learning curve. Perhaps consider taking a second look. In fact, sometimes you will find the learning curve to be pretty reasonable. I recently ran across the book Programming Erlang: Software for a Concurrent World. Once again the Pragmatic Programmers have worked to produce a book that makes learning fun and interesting. The learning curve is manageable, and the examples in the book are more than just trivial exercises. Erlang is an interesting language that makes working in a concurrent environment trivially simple, and on todays multi-core processors, that is becoming important. It can also pave the way to understanding more involved languages like Haskell. It can perhaps even allow you to understand the techniques of functional programming better so you can employ them in your non-functional language more effectively. After all, isn’t being more effective what this is all about?

-Joe