Ok admittedly season 4 had its faults, but I'm not sure it's the worst season of the bunch; for me that was season 3, I was bored senseless!
But it's a subjective thing and while there is a lot of fluff and quite a few episodes that don't hit the mark, the addition of Lois Lane as a semi-regular character breathes new life into the series, and thus some vitally needed witty humour is injected into the often earnest show. It's a joy to watch her making degrogatroy remarks at Clark's expense whilst helping Chloe out with some investigative journalism, I do wonder how they ever did without her. Though I know with regards to the superman mythos Lois was never meant to be in Smallville, but there's no reason one has to accept this as canon and besides Smallville is just one of many versions of the mythos and even the comics have taken liberties at times! So at the end of the day, this season is easier to enjoy if you just accept that Lois is here to stay (especially since she gets added to the credits in season 5!) The writers also do a nice bit of foreshadowing in relation to Clark and Lois's future relationship, which is even more ironic when we see them really not getting along.
Besides the wonderful Lois, I know another problem people had with this season was the Lana-Isobel witch arc. Now I didn't have too much of a problem with it, it was in the background for me thus I could more or less stand it. But I do feel that magic and vampires etc have no place in Smallville, but these writers do tend to take a lot of artistic licensing; so much so at times that I think the producers just literally make it up as they go along... this is evident this season. It's quite obvious they did not have an overall plan for the season, they just went with it and saw what happened; the result was a lot of light-switching and muddled storylines. Some characters did an entire one-eighty within the space of only a few episodes, most notably Lana. But to be fair they were always consistent with her feelings for Clark, so I can't complain about the sudden change too much!
So while the season is a bit confused, the addition of Lois adds some interesting interactions between all the characters. And the search for the stones could have been very good had the witch arc been dropped, as quite often the search for the stones was pushed to background while there were whole episodes devoted to Isobel; due to the wide dislike for the arc it was dropped without much resolve during the finale; not soon enough for some. However, this left a large amount of unanswered questioned hanging over they events of the season... my theory is that when Isobel's purpose was completed and she thought all of those who had her burned at the stake were dead she could move on and her possession of Lana ended. I think it's also easier to assume that she was being controlled by outside forces (Kryptonians?), thus once all the stones were found she did not seek the Fortress of Solitude. It's a complex one, but some bits of the story are detailed quite nicely, like when blood was spilled on the stone (the air stone in this instance) the meteors were called.
Another redeeming factor about this season is the finale, `Commencement' probably the best episode in the history of the show and the extra ten minutes in length help to steady the exciting pace without losing any momentum. The season opener `Crusade' also gets credit for being one of the best episodes in the whole series; with Kal-El taking flight and Lois meeting him in the middle of a thunderstorm (foreshadowing) whilst stark naked (Kal-El that is!)
Other highlights include: the final scene of `Façade', `Unsafe' and `Pariah' which sees the return of Alicia, `Run', `Transference', `Krypto' because its funny, `Sacred' and `Lucy' where Lois's sister comes to town. Worst episodes of the season are `Ageless', `Spell' (though it's not terrible) and `Forever'.
Underrated because the larger faults overshadow some brilliant episodes! Worth buying.