Results that transform businesses

Take a detailed look at the process and the results of an optimisation performed recently by one of our customers.

Baking IT is a SaaS platform for organising, managing and running a baking business - founded in 2011 it already has a considerable user base of 50,000 professional bakers.

Baking IT’s software is incredibly feature rich - it incorporates a 3D cake designer, recipe libraries, a quote generation system, and an insights platform amongst other more fundamental features like contact books, expense and income tracking screens, and a baking-orientated to-do list.

Application performance is obviously crucial for a business like Baking IT, which needs to deliver a great user experience in order to maximise the conversion of people taking advantage of its 14-day free trial into customers, and minimise customer churn.

VITAL STATISTICS

USERS: 50.000
COUNTRIES: 200
CITIES: 7.500
CAKES DESIGNED: 115.000+

BAKING IT BEFORE SKIPJAQ

Prior to working with SKIPJAQ, Baking IT, in common with the vast majority of businesses which transact online, had not attempted to optimise its application stack - i.e. the layers of software running ‘underneath’ its application. In Baking IT’s case this meant that it was using default set-ups for its operating system, web server and runtime.

Tuning the settings in these layers of code is always likely to transform an application’s performance because these layers include the vast majority of the code actually responsible for the application’s overall performance. SKIPJAQ tunes these settings with advanced machine learning techniques; it just isn’t possible to tune them successfully by hand.

UNDERSTANDING PERFORMANCE MEASUREMENT

When measuring performance, it is crucial to look at the whole distribution of latencies (‘latency’ refers to the delay before an application responds to a user’s request) for all of the requests made by all of an application’s users. Here’s what a latency distribution graph for an application might look like:

  • Latency distribution
    • The shape of all the possible request times for an application
    • The width of the distribution shows the range of request times possible for an application
  • Main body (20%-80%)
    • Correspond to the majority of an application’s response times
    • The position and width of the feature determine how uniform and fast the bulk of the requests will be
    • Shifting this to lower latencies, will result in all requests returning faster
  • Tail feature (> 80%)
    • Corresponds to the slowest requests
    • Responsible for the worst user experiences
    • Reducing the size of this tail is of the utmost importance

Within this whole distribution SKIPJAQ always takes particular interest in the ‘tail latencies’ - the worst or slowest group of response times experienced by our customers’ customers. We are especially interested in the 95th percentile latency, which is a particularly useful measurement for describing the worst user experiences delivered by an application. An improvement to this measurement is a sure sign that an application’s overall performance has been improved.

How do you work out what the 95th percentile latency is? Well, if you had 100 response times (typically rendered in seconds and milliseconds), you would rank them from fastest to slowest, and the 95th ranked time would be your 95th percentile latency.

SKIP-TIP
The likelihood that your users will experience a response time from the 95th percentile (or worse) increases when the number of requests they make increases; in fact, after only a relatively small number of requests it becomes a near certainty.

Before bringing in SKIPJAQ, Baking IT’s 95th percentile latency was more than 45 seconds; a significant number of requests were even slower - at over 50 seconds. Response times of this length have the potential to seriously hurt a business, as users click, swipe, or tab away frustrated - possibly never to return.

Overall, the performance of Baking IT’s application, pre-optimisation, was extremely varied, with users experiencing a wide range of response times. Baking IT had much to gain from reducing latency (i.e. running faster) and from offering its users a more consistent experience.

THE OPTIMISATION PROCESS

Baking IT linked its AWS account with SKIPJAQ’s, created an AMI (a snapshot of its application stack), and then SKIPJAQ’s machine learning engine began to search for the optimal configuration for settings in the OS, web server, and runtime layers of its stack.

REMINDER
SKIPJAQ turbocharges the performance of its customers’ applications without changing a line of their code - we do it by finding optimal parameters for stack settings that already exist but are in an untuned state.

SKIPJAQ’s machine learning engine used both the Latin Hypercube method and our proprietary Bayesian optimisation technique (which enables Bayesian optimisation to work out optimal parameters for 50 or 60 settings rather than just 4 or 5) to suggest possible configurations for Baking IT’s application stack. These experimental configurations were launched and tested in SKIPJAQ’s AWS account meaning that the cost of the optimisation process was not charged to Baking IT (an option we offer to all of our customers with a compatible stack).

In a matter of hours, SKIPJAQ’s engine had discovered an incredible performance-enhancing configuration for Baking IT to ‘put into production’. To put that in context, the sun would run out of fuel before a human being, working on the problem manually, would have achieved the same result.

THE RESULTS

When Baking IT applied SKIPJAQ’s recommended configuration to its production stack, the results were immediate and striking. The 95th percentile latency was slashed by over 27 seconds - a crucial improvement of 59%. In addition, average throughput increased by 20%. In other words - Baking IT’s system could handle more users simultaneously and give those users significantly faster and better experiences.

95th percentile latency improvement
27.02
sec
improvement of 59%
Average latency improvement
1.4
sec
improvement of 11.32%
Throughput
improvement
0.66
req/sec
improvement of 20.07%

By making use of SKIPJAQ’s advanced machine learning capabilities, Baking IT was able to optimise its application and deliver an experience that continues to enable its customers to run their businesses in an effective and hassle-free manner, and spend more time doing what they do best - baking!

We’re a rapidly growing business, but we have a small team and most of our time is taken up with adding the new features our bakers need to make their lives that little bit easier. Of course, our customers are also just like everyone else in 2017 - they expect lightning quick website and application performance.

What SKIPJAQ offered sounded almost too good to be true - a huge improvement in performance in exchange for barely any work from us - but they really delivered. After applying the configuration SKIPJAQ discovered to our stack we discovered very quickly that what they say is one hundred percent true: application performance is business performance.

Nishant Raj, co-founder of Baking It