Murphy Choy

Fundamentals to model building? Think about it

In Uncategorized on June 20, 2011 at 11:00 am

A weekend spent SASing away usually yields better results on the blog post. I was entertained by a particular posting online about the fundamental of model building. The forum discussion was about a list of criteria that will guide any beginners about the steps to model building. Let us take a look at them (extracted from the site with editing).

  • Judgment and subjectivity should be minimized wherever possible.
    Models should be supported by enough detailed documentation such that they could be reconstructed by an independent third party and yield identical results.
  • Do not introduce elements to a model simply because they drive results that are more in line with the expectations of the business.
  • Models should be developed by qualified personnel with certified expertise in advanced statistical techniques and under the close supervision of a veteran modeling manager.
  • The model design should be firmly grounded in a generally accepted statistical theory.

These are a bunch of very interesting observations. Most are extremely mathematical and seems to satisfy most stringent academic criteria. However, they seem overly restrictive and may be  counter productive.

The first criteria is very dangerous. Whatever the data tells you must make sense, if it does not, you will need to make a judgement call about the viability of the model. If not, you might want to reconsider the data that has been collected. Subjectivity is always present. Why do you use regression model and not decision tree? That is a subjective call. Perhaps you might use performance measure, but why not the neural network over regression? Too tough to understand? Again, another subjective call. As the modeling process goes on and on, it becomes apparent that subjectivity will creep into the model.

Do not introduce elements that are more in line with business? No problem. The model does not reflect the reality! Well we have to always introduce elements to make the model representative of the business model and understand how external environment influence the business. We should be careful of introducing elements that are not in line with business but the whims and wiles of the business users.

Well a good choice for point three. But can you define an expert? I believe this is very difficult as there are just too many experts out there. Perhaps, one should look at the work one has done. How many models make one an expert? The more the merrier. Certified people may not always be good although certification with plenty of experience will most likely be it.

The last one is a given. Who are we to deny it?

The above opinions are just random rants from a SASer!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: