In case you haven't gotten tired of the various links from Vallen and Jacemora recently, I bring you Alaron of Fluid Druid - and his absolutely excellent, easy to read and understand FAQ on armor pen. It's the best post I've seen on the whole thing so far, and has some really great graphs illustrating how armor pen increases in value as you get more.
Seriously, it puts a lot of the debate to rest and makes it very simple. For now, the rules are easy: if you can get above 600 armor pen, it's the best stat. Otherwise agility is. In 3.2.2, this changes to about 850 armor pen.