From Critical User Journey to SLO/SLIs

We often look at a Service and wonder where to start monitoring the thing let alone what the SLOs should be. Critical User Journeys help to break it down.

https://mobile.twitter.com/googlesre

What is a Critical User Journey

When defining where to start monitoring and what to monitor look at what are your products Critical User Journeys (CUJs).

A CUJ is a flow (or partial flow) that a customer would take through your product that gives the most business value.

Business value could be:

• Financial e.g. the ability to checkout of your store
• Visibility e.g. the ability to share your site on Social Media
• Growth e.g the ability for users to register with your site or accept marketing emails
• Anything else that is important to the success of your product

Defining a CUJ

For example for an online shop the top three CUJs (in business importance order are)

1. Being able to checkout of the store
2. Being able to add items into their basket/cart
3. Being able to browse products

If the customer cannot browse they maybe unhappy but wont have wasted any time and will likely try the site again later. People with items in their carts may still be able to purchase them. However if the customer cannot checkout then they will leave angry at wasting their time browsing and choosing and your business wont make any money

For an online Library the CUJs might be

1. Being able to open a book you have checked out
2. Being able to checkout a new book
3. Being able to return a book
4. Being able to search new books

For a new product or one that is heavily demoed it is worth talking to your sales engineers to see what flows they use for these demos and make that a CUJ too

Actions in the CUJ

Next is to look at what steps are required for the CUJ e.g. for the “Being able to search new books” the actions after a user has hit search might be

1. Web Service checks with ID Service to ensure User is allowed to search
2. Web Service sends search term to Search Service
3. Search Service checks cache for results
4. If not in cache Search Service queries Database (and adds results to the cache)
5. Web Service displays search results

An Architecture diagram for your product will help with this task

Dependancies

From the above actions it can be seen that the CUJ has some dependencies for the journey to be successful. Assuming the application contains the Web Service and and the Search Service the dependencies would be

1. ID Service
2. Cache
3. Database

These could be sub classified as internal and external dependencies depending on if the product team has full control over them or not. e.g. the ID Service might be a totally separate team or company and would be an external dependency where as the cache and DB are owned by the product team and so internal.

What Does Good Look Like?

Now that the actions and dependencies for the CUJ are known, measurable decisions on what a good customer experience is need to be made (SLOs).

Generally for request driven products these metrics will take the form of either:

• Availability — Is the Action or Dependency responding correctly
• Latency — Is the Action or Dependency responding in a reasonable timeframe

Where possible historical data should be used to help define the SLOs and then setting a stake in the ground and tweaking till all parties are happy the goals are achievable.

In the case of this CUJ the SLOs that would go on to form part of an SLA might be:

Latency SLOs are based on successful requests i.e. only requests that return a less that 500 error code. Including error conditions would skew your latency stats

Supporting SLOs can then be defined to help with debugging the application and ensuring all parts of the service are running smoothly

Any Dependencies metric MUST fall with the SLA for the dependency. e.g. you cannot say that the ID Service should return 99% of requests in under 100ms if the SLA for the Service is only 99% in under 400ms

The lower level SLOs have a worst case of 99% in 1.3s (1s from DB search service and 300ms from the ID service) which falls within the Whole CUJ completes SLO of 99% in 1.5s.

Further “long tail” SLOs can be defined if required e.g. it might not be acceptable if 99% of searches using the DB complete in 1s if the other 1% take 30s so another level of SLO could be added. 99% of Search from DB complete in 1s or less 99.9% of Search from DB complete in 1.5s or less

Where To Get The Metrics (the SLI)

Metrics can come from several sources and depending on the Product more than a single source maybe required to gather all the metrics to measure a CUJ. Sources for metrics can also come down to how much development work can be undertaken on the product to expose metrics or extra logging.

As an example for the SLOs defined above the sources could be

Some measure may be available from several sources e.g. the ID Service Latency could be logged and the log file ingested and parsed, could be exposed as a Custom Prometheus metric, could use an APM agent or service mesh etc

Putting it all Together

All the pieces can now be combined into some sort of document to define the CUJ, its SLOs and how the SLIs will be recorded.

If there is no company standard this can be done in anyway that information can be recorded.

e.g. to make a JSON representation

[embed]

Disclaimer This is simply the way I do it. There are probably lots of other ways to break this all down and please do share with me if you have a different way and how you record your CUJs and SLOs etc. Always keen to learn.