The Assessment and Management of Software Products' Procurement Risks

Author(s):

Francois Coallier, P.Eng, CQA
Francois Coallier, P.Eng, CQA, General Manager - IT Procurement, Bell Canada - Acquisitions 2665 Roland Therrien, Longueuil, Quebec, Canada J4N 1C5, (514) 448 5100, fcoallie@qc.bell.ca
Normand Gaudreau
Normand Gaudreau, General Manager - Purchasing Bell Canada - Acquisitions 700 de La Gauchetiere West, Montreal, Quebec, Canada H3B 4L1, (514) 870 2104, gaudreau@post.bell.ca

81st Annual International Conference Proceedings - 1996 - Chicago, IL

Abstract.
The procurement of software products entails risks that need to be assessed and managed. Software products are essentially complex and usually have (and are expected) to evolve during their life-cycle to follow changes in business needs. Also, most software products are bought either with some customization or before their development is completed. This paper presents the approach developed and used by Bell Canada to address this issue. This paper addresses the following: specificity's of software product procurement, risk factors definitions, technical due diligence, risk factors assessment and staffing consideration.

Introduction.
Bell Canada, a provider of telecommunication services with about 60% of the Canadian market, is purchasing for about 1.1 billion Canadian dollars of software based products per year. These include the outsourcing of all of its Management Information Systems (MIS) development and maintenance as well as computer based telecommunication products.

To assess and manage the risks associated with the procurement of these type of products, technical expertise was brought in and developed within Bell Canada's Purchasing organization.

Initially a Software Quality Assurance (SQA) function, this technical arm of the Purchasing organization evolved into a "Quality Engineering" group. This group was then merged with proper purchasing expertise to become a specialized "Information Technology (IT) Procurement organization.

In the last 15 years, Bell Canada has developed processes tools and expertise to address the procurement of software products ranging from complex telecommunication products to management information systems (MIS). Our focus has moved from the classical "quality audits" that were done as part of vendor surveys to context specific vendor capability and product quality assessments focused on procurement needs.

While the traditional supplier quality management function was downsized and integrated in the Purchasing organization, a small group of specialists was put together to address specifically the management of software products' procurement risks.

Concurrently, a risk assessment and management process embedded in its procurement practices were developed and implemented. This process started with the issuance of Request for Quotations and is ingrained in all of the purchasing process, including the management of supplier portfolios.

Specificity of Software Products Procurement. The procurement of software products requires a specific risks management process. This is due to:

  • The high complexity usually associated with these type of products.
  • The fact that, in many cases, the product that is purchased is still in development.
  • Software products are usually expected to evolve as the business and operational environment of the product change.

High complexity is inherent to the nature of software products. This complexity is analogous, in engineering terms, to what is found in large system such as cars. This means that such products are not easy to develop and maintain. Many software products include custom designed hardware.

Often software products are bought while still in development, and/or that some customization is needed. In some case, the product could be mostly custom made (the exception to all of this is shrink-wrap software). This implies that risks associated with the delivery and the quality of the final products can be incurred. This is compounded by the complexity of the product.

Software products need to evolve in response to changes in business environment or practices, or changes in technology. If the product becomes difficult to enhance, significant costs can be incurred. At a certain point, the product may need to be partially or completely re-designed, or replaced by another one. This attribute of a software product is referred to as "maintainability".

All the above implies not only many potential risk elements, but also that the procurement of such product is associated with the establishment of a relationship with the supplying organization.

The procurement of these type of products must include a Due Diligence process that covers adequately the risks associated with these characteristics of software products.

Acquisition Process.
Our acquisition process includes all the activities from the product requirements up to contract management and operation. The following paragraphs describe, in generic terms, the sequence of activities involved during our acquisition process. This acquisition process contains eight steps and is in line with common practices as described in the ISO/IEC-12207 standard.

Need - The first activity is the expression by an internal group of a need for new features or services.

Requirement - The second activity is the documentation of the requirements for the software product to be acquired. This is done by a subject matter expert (or design authority) on behalf of the internal group.

Quotation - Using that requirement document accompanied by general company requirements, a Request For Quotation (RFQ) is issued to the vendor community.

Pre-selection - Using the answer from the vendors, the acquisition agent and the subject matter expert makes a pre-selection based on compliance to requirements and all other business issues. This pre-selection usually narrows down the list to a few suppliers.

Due Diligence - Risks associated with dealing with the pre-selected vendors are then assessed. These include commercial, functional, operational and technological risks. Technological risks are assessed through Capability, Product and/or Project assessments as required. Site visits are usually done for only one vendor.

Supplier selection - The selection of the supplier is a business decision that requires gathering of all pertinent information. All of the relevant issues must be considered . Failure in doing so may results in a less than optimal decision, that could be proven costly during the life of the product.

Contractual agreement - The agreement with the supplier details, the product functionality, but can also cover product improvement goals and process improvement goals. Including them in the contract commitment towards product and process improvement insure the implementation of such programs.

Contract management and operation - This covers essentially the management of the relationship with the supplier. It includes support of internal users, resolution of product and service quality issues, contract updates, monitoring of process and product improvement commitments, etc... Follow-up assessment can be conducted if required.

The scope and effort required by each of these activities depend on the criticality and the nature of the product being acquired. For each new acquisition a customized process derived from our generic acquisition process is defined. This insures that resources are allocated where needed.

Risk factors.
Many risk factors must be assessed to insure a smooth procurement project. These factors can be subdivided as follows:

  • Process
  • Project
  • Product

Process capability factors are related to the practices, human resources and tools used by the supplier to develop and support its product. For instance, the absence of a quality system or proper testing techniques in a development organization is a significant contributor to risks. An improper estimation process will make schedules and resources estimates unreliable.

It is important to note that, while having a positive impact, an ISO 9001 certified quality system does not mean that a developer of software product has the necessary capability to be a low-risk supplier. Such a certification only means that the supplier is consistent in the process and practices that are applied.

Project factors are the constraints put on the development projects such as resources and schedule allocation, clarity and stability of requirements and mastery of the processes used for this development project. An excellent development organization can still miss schedule and quality objectives if given unrealistic objectives.

Product factors are essentially related to the complexity and technical challenges related to the design of the product. Products whose development implies the simultaneous (or concurrent) development of silicon (e.g. IC's), hardware and software are inherently more risky.

While there is a large technical component in the above mentioned factors, organizational and human elements are important contributors. Most software development projects that get into trouble will be mostly because of improper management. Most technical and technological issues can and must be managed.

Technical Due Diligence - As mentioned earlier, our technical Due Diligence is performed doing three types of assessments. These are:

Capability assessment - Capability assessments are done using, as a benchmark, our TRILLIUM model (a derivative of the Software Engineering Institute's Capability Maturity Model). This model is a collection of "best in class" software product development and support practices. Such assessments help us in evaluating the development and support capability of a the supplier for a specific product line. These assessments are highly focused on the product being acquired.

Product assessment - A product assessment is conducted if the product is critical or if the capability assessment indicates at least a medium or a high risk level. The product assessment aims principally to assess the maintainability of the software product as defined by the ISO-9126 standard [ISO-9126]. The product is thus assessed from an architectural, a design, and an implementation point of view. For those assessments, we go as far as doing a complete analysis of the source code in some cases. We have co-developed with a university a method and its associated tool that make such an assessment practical.

Project assessment - A project assessment is done when the product is under development. These assessments are done to evaluate the probability that a given product will be delivered with all its defined functionality and within schedule. These assessments are conducted on a regular basis during the development life cycle of the product.

Risk factors assessment.
Assessing the risk factors mentioned previously is not a trivial undertaking. Not only does it require appropriate expertise (like for any type of Due Diligence), but it must be done under the schedule constraints of the procurement activity and the limitations associated with any auditing activities.

The most important resource needed to perform such a risk assessment activity is proper staffing. The personnel we use is an expert in either Management Information System (MIS) or telecommunication & software engineering with a graduate (Master) degree and at least 5 years of product development and auditing experience.

It is very important that the personnel we use be not only credible internally, but also for the vendors. Care must be taken in the selection of this personnel since on top of the technical experience it must have the managerial and interpersonal skills required from professional auditors.

A critical mass of such people is needed in a procurement organization to perform such a function. This is needed to insure the appropriate span of expertise and also peer review. Many assessments will need to be done by two experts to insure proper diagnostic and also to be able to do it in the short amount of time available on the vendor premises (One to two days usually). Also, proper resources must be available to maintain and expand the expertise of this personnel.

Conclusions.
Our processes and methods have been in constant evolution in the last three years, and we expect this to continue as we acquire more experience. Specifically, we must always fine tune our processes to adapt to the procurement of new type of products such as ATM and Broadband technologies or client-server based systems.

We have found that having such a capability in a purchasing organization is an asset as long as proper focus is maintained. The deliverable of a technical Due Diligence exercise is not a technical report but a diagnostic that can be used for decision making.

REFERENCES

Anonymous, 1991, ISO/IEC 9126, "Information technology - Software product evaluation - Quality characteristics and guidelines for their use", ISO/IEC.

Anonymous, 1995, ISO/IEC 12207, "Information technology - Software life cycle processes" ", ISO/IEC.

Coallier F., 1994, "How ISO 9001 Fits Into the Software World ", IEEE Software January 1994 pp. 98-100.

Coallier F., McKenzie R.M., Wilson J.F. and Hatz J., 1995, "TRILLIUM - Model for Telecom Product Development & Support Process Capability ", Bell Canada (available on the World Wide Web at http://ricis.cl.uh.edu/trillium/).

Jones C., 1994, " Assessment and Control of Software Risks ", Yourdon Press computing series.

Mayrand J. and Coallier F., 1996, " System Acquisition Based on Software Product Assessment ", to be published in the Proceedings of the 18th International Software Engineering Conference, Berlin, March 1996.

Paulk M.C., Curtis B., Chrissy M.B. and Weber C.V., 1993, "Capability Maturity Model for Software ", version 1.1, CMU/SEI-93-TR-24.


Back to Top