Vendor due diligence
So, if buying from a very large vendor, it will be easy enough to look at their general health because their finances are public (OK, there are sometimes accounting frauds but you can't do much about these). The usual ways of analyzing company's health can be used here. I highly recommend the "Guide to analyzing Companies" by the Economist, which is clearly written and gives excellent examples of the indicators of company health and also of early warning signs. You should not assume that just because a company is publicly listed then it must be OK - ask the customers of Commerce One about that approach, for example.
In such cases you will probably find that the company is fine, so the bulk of the diligence efforts should instead be directed to how important this particular product is to the company, and so assess how likely is to to get ongoing development. Of course the vendor is hardly a reliable source here, but you can seek advice from analysts, and also it is fair to ask the vendor
just how many customers of this particular product there are (you should be able to talk to some of them). For example, SAP's MDM product had seemingly shifted just twenty copies into customer use throughout its 18,000 strong customer base in two years. Given this dismal penetration rate it was perhaps not a shock that they dropped it (their replacement MDME offering is based on an acquisition of PIM vendor A2i). Anything which is not contributing to a vendor's sales figures on any scale should be considered suspect.
In the case of small vendors you have different problems. You can be pretty sure that the product is important to the vendor, since it is probably the only one that they have. The question is whether the vendor will survive. This is trickier, since the company is probably privately held, and so is not obliged to publish its accounts, at least in the US. In the UK you can get around this by paying a few pounds to Companies House and look up their acocunts.
If you are making a large purchase then it is fair game for you to ask for information on the company financials, and you should get nervous if they refuse. One thing that will achieve little is to ask for a letter from the VCs backing the company. They will inevitably sing its praises; they are hardly going to say "ah, this one's on a bit of a knife-edge, I'd watch out if I were you". Indeed I knew of one case where a major deal was in progress at a BI vendor, and through a contact I became aware that the entire future financing of the (cash strapped) company was dependent on this deal going ahead; in such cases you cannot expect objectivity from investors.
So, what can you do? Well, profits are an opinion but cash is not. Hence, assuming you can see some figures, you can get a sense of how much cash the company has, and attempt to work out the "burn rate" i.e. how fast are they burning through this cash (most VC backed start-ups are unprofitable; if they were profitable then they probably wouldn't need expensive VC money). However this on its own may give false signals. Due to their IRR-driven instincts, VCs don't dole out more cash than they need to start-ups; they like to always have the option of pulling out if they need to, so it is rare for a start-up to actually have more than about one year's cash needs in hand. The question is: will they be able to raise more cash if they need it? This is a complex subject, but essentially you should be able to get a sense of this by talking to analyst familiar with the VC community. For example, companies growing at 50% or more are very likely to be able to raise cash, even if they very unprofitable. The gross margins in software are commonly 90%, so profits will come eventually if the company can just grow large enough; this is why VCs invest in software companies more than, say, restaurants. So if you cannot find someone knowledgeable to look the figures over for you and make an assessment, then a decent proxy for security is the revenue growth rate. If the company's growth is stalling (say 10% a year growth for a small-medium software company) then things could be sticky in a future financing round. This is a generalization (and companies with a subscription model, for example, have a much more predictable life than ones selling traditional licenses) but it may be the only real set of figures that you can dig out.
Another source of due diligence is other customers, who may well have done exactly the same due diligence exercise as you fairly recently. Of course you have to be careful it was not out of date, and you should check how thorough they really were, but you may be able to save yourself a lot of work. If three Fortune 100 companies recently did detailed due diligence on a vendor and bought its software anyway, this may help you at least feel better.
Remember: the company or product does not have to be around in ten years if your payback case is 13 months. The faster the payback period for the product, the less concerned that you need to be about agonizing over the long term future of the company, or of the product within the vendor. You did do a proper business case for the purchase, right? Of course you did.