How to measure success

You might think: ‘What kind of a question is that?!’ or maybe even ‘What does this have to do with Internet technology or software?’ So, I want to specify this question right at the beginning: ‘How do we measure the success of self-care and self-service software?’ I’ll try to deliver an explanation in this article and concentrate on the question what success means, especially for an operator or ISP, to build a sound basis for the following blog post in which I will focus on the ‘How’.

What is success?

In my role as a business analyst I’d like to start with a definition of the term in question; what is “success”? The online encyclopaedia “Wiktionary” provides the following description: Success is the “achievement of one’s aim or goal.” The Oxford Dictionary explains this term as an accomplishment of an aim or purpose / the achievement of desired aims. So success is:

  • a result that has been aimed for
  • a result that has been set individually beforehand

The success of a cause or action is therefore determined by the goals that have been set in advance. This leads to the conclusion that success is subjective.

How does this fit into the measurement of success? Well: subjective characteristics are hard to measure, since they do not offer universal key figures. We are all familiar with the saying: “Beauty is in the eye of the beholder” – the same goes for “success”: it’s an individual value. For the measurement of the success of software this means that the measurement values need to be defined based on occurrences and this definition needs to happen already during the conception of the software, so that the definition fits the actual implemented software functionality.

Measurement of success and causality

In practice this means: To measure the success of our software we have to define our goal, so that we can pursue this goal. In our case the main goal would be: our software should help operators and ISPs to avoid unnecessary costs for customer support – therefore to reduce costs. So if there is a reduction in the service costs after the usage of the software over a time period A in comparison to the reference period B, the software would be a success. However, this is not as easy as it seems: according to this definition our software would also be successful if in time period A half of the ISP customers switched to a competing operator/ISP and as a result only half of the customers called the support hotline in comparison to the reference period B. Our simple definition is therefore not as easily applicable. The reason for this is the missing specification of cause and effect, the causality. Instead of expanding our definition with lots of limitations and conditions the key for this dilemma is much easier: divide our high level goals into smaller, better measurable goals.

  • Our software should inform users proactively about problems on their PC and repair the error to prevent the user from calling the service hotline.
  • Our software should help users with the installation of their modem to avoid a service technician appointment.

In order to reach our main goal, in both cases users should not contact the ISP but rather master these tasks by themselves with the help of our self-care solution: the cost reduction in the service sector. Therefore, it’s all about the hard facts: How many hotline calls have been prevented, how many service technician hours have been saved? In short, how can we measure events that have been prevented and therefore de facto never happened?

Well I want to conclude this article with an answer for this question: not at all. It’s just not possible to reliably measure events that never happened.

But before you get upset about the possible waste of time reading this article, let me reassure you that it was not in vain! Even when we concluded that it is not possible to measure events that never happened, there are alternatives. And in the next article I want to present those alternatives to explain the “How”, as mentioned in the beginning of this article.