Follow us...

Enterprise Software Selection Due Diligence

Tom Bellinson, ITHAKA
 
I recently presented the “Selecting & Preparing for the Implementation of a New ERP System” webinar for IIBA® Members, and a lot of questions came up around information gathered during the due diligence portion of a software selection project. There are a lot of moving parts to selecting a good enterprise software product, and careful upfront evaluation is needed to help make the right choice.
 
Due diligence is a way of objectifying the validation necessary to take the final step. The more structured it is, the more likely you will make the right selection. It is important to remember that there are never any guarantees that your choice will be perfect. That is why I have a well-defined set of activities associated with due-diligence. They are:
  1. Vendor background
  2. Vendor organization
  3. Vendor’s client feedback, including:
    1. Telephone
    2. On-site
 
The first two items can be gathered in one step. I recommend two steps for the third item. Let’s take each in turn.
 
Vendor Background & Organization
 
This is the process of searching for skeletons in the closet. Is the vendor financially solvent? Are they organized? Employers routinely run credit checks and criminal background checks on prospective employees. You should do no less for an enterprise software vendor. Dun & Bradstreet provides services which offer some of this information. You may be able to search online for court records to see if the company is involved in any litigation. It might be nice to know if they have any patent infringement lawsuits pending—or worse yet, customer class action suits.
 
Here is a list of things you should know about a vendor. We will take a look at each of these in more detail:
  1. Company background (vital statistics)
  2. Staff levels
  3. Software development cycle
  4. Testing procedures
  5. Feedback mechanisms
  6. Training options
  7. Product roadmap & strategic decision-making process
  8. Industry acknowledgements
  9. Financial stability
  10. Legal issues
  11. Pending company sale/merger prospects
Company Background
Unless the firm you are considering is an industry stalwart, you should make sure you understand who you are dealing with. Some details to gather are:
  • Number of employees
  • Years in business
  • Annual revenue (private companies may be reluctant to give specifics, but get a ballpark)
  • Company ownership structure (private vs public, number of owners, LLC, Corporation, etc.)
  • Number of active clients (currently using their software)
  • Headquarters location plus satellite offices
Staffing Levels
Depending on the type of product you are purchasing, you may find yourself very dependent on the services of the company. It is important to understand certain ratios of staff to clients. Think about the interactions you might have with the company and determine which of the following groups will need to be robust enough to handle the load effectively:
  • Inside support staff
  • Implementation consultants/project managers
  • Software developers
    • New product development
    • Customer requested customizations
  • Software testing staff
  • Salespeople
  • Administrative staff
Software Development Cycle
Traditional software development is done by gathering requirements, writing detailed specifications, building the software, testing it, and deploying it to production (possibly initially on a limited basis). Newer agile techniques have combined these steps into smaller increments or iterations. Some companies shorten the entire cycle, while others only shorten parts of it. In other words, they may iterate on the first few steps but still deploy new features to production less frequently.
 
Understanding how a company does development, what tools they use, and what methodologies they employ will allow you to get a sense of how long you will need to wait for enhancements that are on the product roadmap. Some companies have a set number of releases each year, while others may be doing what is called “continuous delivery”—where features may appear one at a time as soon as they are tested and deemed ready for primetime. 
 
It is also important to discover how companies manage customizations. Do they have separate teams for these, or are they incorporated into the main product development workflow? How is this work prioritized? Does money talk, or does the product roadmap reign supreme?
 
Testing Procedures
When you are trying to run a business, buggy software is never a good thing. Enterprise software is complex as it has many components which are highly interactive. Even if a new module works by itself, it can have unintended consequences in other parts of the system. Well-designed software will have clearly defined interaction maps that tell developers which parts of the system will be affected by any changes they might make. This also informs the testing team so that they can ensure the new functionality is fully operational and has had no adverse effects (otherwise known as “regression” in developer speak).
 
When writing test procedures, it’s helpful to keep these questions in mind:
  • How many people test a new release?
  • How is testing structured?
  • Do they use test-driven development?
  • Do they have unit tests in their code, and if so, what percentage of it is covered by unit tests (if they say 100%, they are probably lying -- this is extremely difficult and arguably a waste of time to do)?
  • Do they use automated user interface testing tools?
  • What other techniques are being used to ensure the system is fully functional before deployment to production?
This is also a good place to find out about the last time they had a bad release. Almost everyone has one somewhere along the way. It’s helpful to note:
  • What went wrong?
  • What did they do to mitigate it with their clients?
  • What did the do to ensure it never happens again?
 
Feedback Mechanisms
The mandate of most enterprise software companies is to keep their clients happy. The best way to do that is via communication. With a software company, that means not only giving clients multiple outlets to communicate with the vendor, but also allowing clients to talk amongst themselves. When an organization lacks a channel to allow clients to talk with each other, this can be a bad sign indicating that they are hiding something.
 
Here in the 21st century, social media is a common way to allow people to talk with the vendor and each other. Many vendors have forums on their website as well. Newsletters (either paper or email) help keep clients informed about what the vendor is planning and what they have done recently. Annual user conferences can be an excellent way to network with the vendor and other clients. Vendors often run special classes at these events and even offer discounts on new modules to attendees. If your vendor holds these, you should consider going or sending someone from your organization. If you are buying new software and one of these classes is upcoming, you can’t get any luckier than to be able to attend (if they’ll let you—some won’t until you buy).
 
Take an inventory of the different ways the vendor monitors clients and clients can monitor the vendor. Are they more or less interactive? Interactivity is good, but more is, too. The feedback mechanisms (and which way they predominately flow) will tell you a lot about the personality of the vendor.
 
Training Options
Options for training generally include:
  1. Classroom instructor-led classes
  2. On-site instructor-led classes (or one-on-one training)
  3. Online tutorials
  4. 3rd party training options
  5. Disc based training (old-school)
  6. Self-paced printed study materials (also old-school)
  7. “Train the trainer” training
Be particularly wary of companies that have an overreliance on #7. “Train the trainer” programs work well if you have in-house training staff, but you should at least assess if you have good educators within your ranks. It shouldn’t be assumed that they are by default.
 
Product Roadmap & Strategic Decision-Making Process
Ask to see a product roadmap. If the company doesn’t have one, it’s usually a bad sign. It often indicates that they are completely reactionary, which means the highest bidder gets their work incorporated.
 
A good product roadmap should indicate the sequence in which new functionality will be added, along with a rough sizing of the initiative. You will want to understand how things get added to (and removed from) the roadmap. Does the roadmap indicate any end of life plans for the product? Has the vendor sunsetted other products? If so, how recently?
 
Industry Acknowledgements
Does the company have any awards? If they specialize within a single industry, being recognized by that industry is a good sign that they have positive recognition amongst others in your line of business (or your client’s). A lack of awards or acknowledgements isn’t necessarily a bad thing, but it is noteworthy.
 
Financial Stability
I already mentioned about using Dun & Bradstreet or some other credit rating service to identify the vendor’s financial strength. It’s good to understand how the firm is backed financially. Are they publicly traded? Do they receive private equity funding? Are they completely self-funded? If it is the latter, it is especially important to determine their financial stability. What are your options if they go out of business?
 
Legal Issues
Any pending legal actions against the company should be fully disclosed, as well as any actions the company is taking against another entity. Legal actions are not only a distraction, but they also drain company resources that could be used for things that more directly benefit clients.
 
In addition to pending or active legal actions, try to get information about past issues. Few settlements are sealed, so any complaint that required judicial review will have a record somewhere. If the issues involved one or more clients, find out what changes have been made to avoid similar situations in the future. Again, whether the vendor is prosecuting or defending a case, it is still cause for concern.
 
Pending Company Sale/Merger Prospects
For a time, there was major consolidation in the enterprise software industry. Big fish were gobbling up little fish by the dozen. Companies come together usually for one of  two reasons:
  1. They want to add to their product offerings
  2. They want to build their customer base
If the vendor you are buying from is smaller, they could be an acquisition target. If they have one or a few owners, they may be looking to cash out. If they are bigger, but their product is older, they may be looking for a technology upgrade. Whatever the circumstances of a sale or merger, you will want to have some assurances that the product you are buying has a future. Further, you will want to know that the relationship with the vendor is not going to suddenly change—for the worse!
 
Armed with all this data, you should have a solid picture of what the company looks like and whether they are a good fit for your organization. There! Check step #1 and #2 of the due diligence process off your list.
 
Vendor’s Client Feedback
The best way to find out how you will be treated as a client is to talk with your chosen vendor’s other clients. If you know someone in your industry who is using their software, they can be a great source of information. However, as it turns out, getting good client feedback is challenging. There are three main reasons for this:
  1. People are generally nice and don’t like bad-mouthing others
  2. People have selective memories and try to forget bad experiences
  3. Vendors rarely share the names of their implementation disasters with prospects
In an ideal world, a vendor would give you a complete client list and allow you to choose who to contact. This happens, but is extremely rare. Generally speaking, the smaller the list of reference clients, the more likely the company is hiding something. The vendor may offer you a small list of names of companies pre-filtered either by geography (nearby) or industry (comparable usage). I suggest you do not accept these filters. Ask for the whole list, and contact clients both near and far, as well as within your industry and in other industries (if available).
 
Also, ask if reference clients are compensated in any way. Some vendors allow good reference accounts to gain early access to new functionality or to be on focus panels used to decide the product roadmap. Others could even be given free support or even free software. I have never come across a client that was paid outright for a reference, but I would not assume it has never happened. The point here is that you should understand the relationship the client has with the vendor so that you can properly assess the feedback you receive from them.
 
Telephone Interview
Even if a client is being paid outright for their participation in the reference process, it doesn’t mean that you can’t get good feedback. It’s important to ask the right questions. Here are some of the questions that I ask (you should think of others that are specific to your needs):
 
  • In what year did they purchase the software?
  • How many users are there?
  • Are they using the current version? If not, how many versions behind are they and why?
  • If in-house, what hardware they are using to deploy the application?
  • Which modules are they using?
  • What was the biggest implementation challenge?
  • What is the vendor’s biggest shortcoming?
  • Which feedback mechanisms do they use? How are they?
  • How good is the support received from the vendor?
  • What is the biggest user complaint about the system?
  • If you could change one thing about the system or vendor, what would it be?
  • What was the biggest problem they have ever had with the system, and how was it handled?
  • What are the top three specific benefits of this system over the last one?
  • If using any 3rd party add-ons, what are they and what is their quality?
  • What are the effects on quality since implementing the system?
  • What are the effects on costs since implementing the system?
  • Have they noticed any effects on customer service?
  • What is the one piece of advice they can give you going into this implementation?
Many of these questions may make it seem like you are fishing for bad news. You are! You probably won’t need to fish too hard for good news. Put yourself in the reference client’s position: unless you hate the system and are completely dissatisfied with it (which would likely disqualify you for this task anyway), you benefit from the vendor adding a new client. The vendor gets more revenue, which will likely help their financial stability and ability to add resources.
 
To get good information, you may have to talk to more than just the carefully vetted staff member of the client company—especially if that person cannot provide detailed answers. See if the vendor’s client will let you talk with others from the company. Generally, if you can talk with two or more people from a single reference account, you are likely to be getting a more accurate picture of how the product is really working for them.
 
On-Site Reference Client Visit
It’s highly recommended to witness the system in action at a client’s facility. These visits are often highly controlled affairs in which a vendor representative (usually your salesperson) will be accompanying. You should try to bring as many people as possible to this. If you outnumber your hosts, you can often have members of your party hang back and get feedback from regular users while the group has begun to move on to a different area. This is a little sneaky, but some of the juiciest information can be garnered this way. If you are expressly asked not to do this, then I wouldn’t suggest it, but I’d also make note of the fact that they are trying to control the conversation.
 
More importantly, seeing how someone else is using the system can inform your own plans. Maybe you were told the system works a certain way, but you see that the client isn’t using it that way. Why? Maybe they tried it and had to take a different path for some reason. Wouldn’t that be good to know before you go down the wrong path yourself?
 
Ask a lot of process questions when you’re on-site. This is a chance to learn if the system will actually support your processes or if there are reasons why it can’t and what the potential workarounds might look like. These visits are a give-and-take affair so the more generous you are with how you do things; the more likely they will be to share their approaches.
 
Finally, keep your ears open. You might overhear a conversation that speaks directly to the system. A lot, more than knowledge of the system’s viability for your organization can be garnered from these little excursions.
 
Conclusion
Since software is not static, its evolution is of critical importance to your organization. You are far better off picking a second-choice product that is on a solid trajectory than a first-choice that just happens to be passing through your neck of the woods on the way to somewhere else that you don’t want to go.
 
This may seem like a lot of work after having labored for months selecting a product that best fits your processes and user culture. That’s because it IS a lot of work. It is work that is well worth the effort. If you approach this process as building a relationship with the vendor rather than simply making a purchase, you will put your whole organization in the right frame of mind.