The navy blazer is commonly considered the most versatile kind of sport coat you can own, and for good reason. You can pair with almost any kind of trousers, and if made in the right weight, it can be worn year-round. Or, if your climate is less temperate than ours in California, consider getting something in a mid- to heavyweight wool (like a hopsack weave) for the fall and winter seasons, and then a lighter weight tropical wool for the spring and summer months.
Although this is pretty good, standard advice, I’ve long thought that the most useful color for sport coats is actually brown. Especially for this time of year. Think of dark brown or golden tan corduroys, or any kind of tweed – solid brown Shetlands, fuzzy herringbones or houndstooths, or any of the checks you see in the first photo above (which is of the Duke of Windsor’s closet, who was a very daring and dandy dresser, but also know the value of sedate brown coats).
In fact, if you don’t want to have a massive wardrobe, you could get along just fine with one or two navy sport coats and a couple more in brown.
For a little more variety, however, my friend Andrew Yamato recently wrote a good post about fall jackets at A Suitable Wardrobe. As he notes, fall justifies the bringing out of colors such as rusty reds, burnt oranges, slate blues, ochre yellows, and mossy olives. I think olive sport coats are particularly useful in the fall – quiet and conservative enough for men who don’t want to stand out too much, but also a nice deviation from the usual colors most of us wear.
Pair these with ancient madder ties, chunky sweaters, suede shoes, and fuzzy flannel trousers. Ribbed corduroys and cavalry pants will also work well, if you want to expand your wardrobe in that direction. Otherwise, chinos make for a nice three-season, spring-through-fall alternative. Shirts should probably be light blue oxford cloth button downs, not just because they have the right visual weight to hold their own against heavier sport coats, but also because I think they’re the best kind of shirts.