How We Improved the Performance of B2B eCommerce
Platform by Data Caching in Azure Cloud
This tech-related article is about our experience of caching a Virto Commerce application written in ASP.NET Core and running in the Azure Cloud. Commonly, caching can significantly improve the performance and scalability of heavy applications, including ecommerce platforms, by speeding up access to data from the back end. Technically, caching works better with data that changes relatively infrequently and/or is expensive to create.
There are only two hard things in Computer Science: cache invalidation and naming things.
— Phil Karlton, Netscape Architect
The Virto DevLabs team tested several different ways of caching data in the Virto Commerce app in order to reduce the load on external services and the database, also to minimize the application latency when processing API requests.
Below we’ll talk about the technical details of caching methods, which we rated as the best ones and that we use on our B2B ecommerce platform. Here is a list of the topics covered in this article:
● Challenges raised when using distributed cache;
● Exclusive “thread-safe” retrieval of data for adding to the cache, with multithreaded access;
● An original approach to generating keys for caching, especially for objects of complex types;
● Managing the lifetime of objects in the cache through global settings and logical associations into regions;
● Caching null values;
● Synchronization of local caches for different instances of the application through a third-party service (backplane).
Cache-Aside Pattern Overview
We chose Cache-Aside as the main pattern for all caching logic because it is very simple and straightforward for implementation and testing.
The pattern enables applications to load data on-demand:
Cache-Aside works according to the classic behavior: when we need specific data, we first try to get it from the cache. If the data is not in the cache, we get it from the source, add it to the cache, and return it. Next time, this data will be returned from the cache. This pattern improves performance and also helps maintain consistency between the data stored in the cache and the data in the underlying data store. For more information on the Cache-Aside pattern, see the Microsoft documentation.
Cache-Aside Challenges
This section is about the challenges of implementing the Cache-Aside pattern using ASP.NET Core in-memory cache. We did not use a distributed cache in the platform code because we wanted to keep the platform configurable and flexible, and prefer to address potential scalability issues in other ways (see Scalability section below).
The distributed cache has three significant drawbacks that influenced our decision not to use it in our B2B ecommerce platform:
● All cached data must support serialization and deserialization, which is not always possible to do transparently for all entities in the application;
● Performance degradation is possible compared to in-memory cache due to network calls for cached data (network latency);
● Since it is not always possible to abandon the in-memory mode when using mixed caching (in-memory + distributed) it will be difficult to determine which type to use at a particular moment. This is especially important for open-source products, and for products like Virto Commerce, as B2B eCommerce platform will not be a plus definitely.
Serialization converts an object to a stream of bytes, so it can be taken out of a process, either to be stored or to be sent to another process. Deserialization is the reverse process that converts a stream of bytes back to an object.
The serialization provided by the .NET framework has two main drawbacks:
Slow: .NET serialization uses reflection to validate type at runtime. Reflection is a slow process compared to precompiled code.
Inefficient: .NET Serialization stores the fully qualified class. Name, assembly details, and references to other instances in member variables, all of which makes the serialized stream of bytes much larger than the original object.
Therefore, for caching Virto Commerce B2B platform, we chose to work with an in-memory cache. It should be said that ASP.NET Core supports distributed caching, but we decided not to use it due to the above-mentioned drawbacks.
A simple Cache-Aside pattern implementation using the IMemoryCache abstraction looks like this:
public object GetDataById(string objectId)
{
object data;
if (!this._memoryCache.TryGetValue($”cache-key-{objectId}”, out data))
{
data = this.GetObjectFromDatabase(objectId);
this._memoryCache.Set($”cache-key-{objectId}”, data, new TimeSpan(0, 5, 0));
}
return data;
}
This basic implementation of the code still has several disadvantages (and, therefore, we will present our solution below):
● It requires the manual creation of a cache key and knowledge of the rules on how to create keys, plus consider their uniqueness;
● It is not “thread-safe,” i.e., multiple threads will try to access the same cache key at the same time, which can lead to redundant access to the data source. This may not be a problem unless your application has high concurrency and costly server-side requests; and
● Manual control over the lifetime of the cached data is assumed. Choosing the right values for the lifespan is difficult and reduces developer productivity, and also makes it difficult to maintain the application due to complex settings for the lifetime of objects in the cache for various application subsystems.
The relatively new MemoryCache methods, GetOrCreate / GetOrCreateAsync, also suffer from these drawbacks, which means that we cannot use them as they are. This article describes the problem in more detail: ASP.NET Core Memory Cache — Is the GetOrCreate method thread-safe.
What We Did to Improve the Code
To solve the above-mentioned drawbacks, we defined our own IMemoryCacheExtensions. This implementation ensures that cached delegates (cache misses) are called only once, even if multiple threads are accessing concurrently (race conditions). In addition, this extension provides a more compact syntax for client code.
Here’s a variation of the previous code example with a new extension:
1 public object GetDataById(string objectId)
2 {
3 object data;
4 var cacheKey = CacheKey.With(GetType(), nameof(GetDataById), id);
5 var data = _memoryCache.GetOrCreateExclusive(cacheKey, cacheEntry =>
6 {
7 cacheEntry.AddExpirationToken(MyCacheRegion.CreateChangeToken());
8 return this.GetObjectFromDatabase(objectId);
9 });
10 return data;
11 }
Generating Keys
The special static class CacheKey (line 4 in the example above) provides a method to create a unique string cache key according to the passed arguments and type/method information.
CacheKey.With(GetType(), nameof(GetDataById), “123”); /* => “TypeName:GetDataById-123” */
CacheKey can also be used to generate cache keys for complex object types. Most platform types derive from the Entity or ValueObject classes, each of which implements the ICacheKey interface, which contains a GetCacheKey () method that can be used to generate cache keys.
In the following code, we created a cache key for a complex type object:
class ComplexValueObject : ValueObject
{
public string Prop1 { get; set; }
public string Prop2 { get; set; }
}
var valueObj = new ComplexValueObject { Prop1 = “Prop1Value”, Prop2 = “Prop2Value” };
var data = CacheKey.With(valueObj.GetCacheKey());
//cacheKey will take the value “Prop1Value-Prop2Value”
Thread-Safe Caching and Avoiding Race Conditions
Now let’s talk about thread-safe caching:
In line 5 (in the example above), the _memoryCache.GetOrCreateExclusive () method calls the thread-safe caching extension. This ensures that the cached delegate (invoked when no data is in the cache) is executed only once when called concurrently from multiple threads.
An asynchronous version of this extension method is also available: _memoryCache.GetOrCreateExclusiveAsync ().
The following code demonstrates how this exclusive access to a cached delegate works:
public void GetOrCreateExclusive()
{
var sut = new MemoryCache();
int counter = 0;
Parallel.ForEach(Enumerable.Range(1, 10), i =>
{
var item = sut.GetOrCreateExclusive(“test-key”, cacheEntry =>
{
cacheEntry.SlidingExpiration = TimeSpan.FromSeconds(10);
return Interlocked.Increment(ref counter);
});
Console.Write($”{item} “);
});
}
The console output will be like this one:
1 1 1 1 1 1 1 1 1 1
Cache Expiration and Eviction
The management of the lifetime of objects in the cache occurs on line 7:
cacheEntry.AddExpirationToken (MyCacheRegion.CreateChangeToken ())
where the CancellationTokenSource object is created. It is associated with cached data and the strongly typed MyCacheRegion region, which allows you to combine cached data into logical groups (regions) through which you can manage the lifetime of these objects in the cache. For example, remove all associated data from the cache with one simple call to the MyCacheRegion.ExpireRegion () method. Learn more about how cache dependencies work in the ASP.NET Core Memory Cache dependencies article.
For in-memory cache, there is one mechanism in the cache-dependencies that allows you to implicitly associate cached data into one group. Here’s an excerpt from the documentation:
using (var entry = _cache.CreateEntry(CacheKeys.Parent))
{
…
}
With the using pattern in the code above, cache entries created inside
the using block will inherit triggers and expiration settings.
Thus, all data that will be added to the cache inside the using section will inherit the lifetime settings from the parent. In this project, we abandoned such a hidden feature, since it led to unpredictable behavior for managing the lifetime of objects in the cache, and it took a long time to catch the bugs.
If we talk about managing the lifetime of objects in the cache, then in our product we avoided manual control of the lifetime of cached data in the code. The Virto Commerce platform has a special CachingOptions object that contains the absolute or relative lifetime settings for all cached data (see below).
Strongly Typed Cache Regions
Our platform supports cache regions, which are used to manage the lifetime of grouped / linked objects in the cache.
For these purposes, we have defined a generic class CancellableCacheRegion <>. This class has the AddExpirationToken and ExpireRegion methods that can be used to add or remove all data from the cache for a given region:
//Region definition
public static class MyCacheRegion : CancellableCacheRegion<MyCacheRegion>
{
}
//Usage
cacheEntry.AddExpirationToken(MyCacheRegion.CreateChangeToken());
//Expire all data associated with the region
MyCacheRegion.ExpireRegion();
There is also a special GlobalCacheRegion region that can be used to expire all cached data for the entire application:
//Expire all cached data for entire application
GlobalCacheRegion.ExpireRegion();
We were surprised to find out that the typical implementation of InMemoryCaching lacks a built-in way for clearing the entire cache, so the method with the global region had to be implemented by ourselves, as recommended in this article https://stackoverflow.com/questions/34406737/how-to-remove -all-objects-reset-from-imemorycache-in-asp-net-core.
Subsequently, we got into trouble with this approach due to the fact that the tokens associated with the global region were not cleaned up by the garbage collector and led to huge memory leaks. Here is the Gist that demonstrates the problem: https://gist.github.com/tatarincev/4c942a7603a061d41deb393e0aa66545.
As a result, this problem was successfully solved.
Caching Null Values
By default, Virto Commerce B2B ecommerce platform caches null values. If negative caching is a design choice, this default behavior can be changed by passing false to cacheNullValue in the GetOrCreateExclusive method, for example:
var data = _memoryCache.GetOrCreateExclusive(cacheKey, cacheEntry => {},
cacheNullValue: false);
Cache Options
We have moved the three main cache management parameters into the application configuration so that these parameters can be changed depending on the operating conditions.
appsettings.json
“Caching”: {
//Set to false to disable caching of application data for the entire application
“CacheEnabled”: true,
//Sets a sliding expiration time for all application cached data that doesn’t have an expiration value set manually
“CacheSlidingExpiration”: “0:15:00”,
//Sets an absolute expiration time for all cached data that doesn’t have an expiration value set manually
//”CacheAbsoluteExpiration”: “0:15:00”
}
Cache Scaling for B2B eCommerce Platform
Now a few words about the previously promised scaling. Running multiple platform instances, each with its own local cache, and which in turn must be consistent with the cache of other instances, can be challenging. Without solving this problem, each instance of the application will have inconsistent data, which will definitely not be liked by Virto Commerce clients.
To solve this problem, we needed some kind of external service that would act as a backplane for all operations to remove objects from the cache, and to which all instances of the application would be connected. Through pub/sub, instances would report or learn about all changes to the cache in other instances, thereby keeping the cached data in memory consistent with the actual values in the data source.
The How to scale out B2B ecommerce platform on Azure describes the process in more detail. It also shows how to configure Redis as a backplane for synchronizing local caches in the case of multiple instances of the Virto Commerce e-commerce platform.
Conclusion
In this article, we have introduced you to some of the most common caching implementations and issues that come with it. At the same time, we talked about how we dealt with these problems in the Virto Commerce .NET B2B eCommerce platform project.
Use IMemoryCacheExtensions, which contains sync and async methods as a compact form of implementing the Cache-Aside pattern in the ASP.NET Core IMemoryCache interface. These extensions provide exclusive access to the original data in a race condition.
To avoid problems with stale cached data, always keep cached data in a consistent state. Use strongly typed cache regions that allow you to delete groups of data. By the way, Virto Commerce B2B ecommerce platform uses an aggressive caching policy for most DAL services, even caching large search results.
Thank you for reading this article. If you have any questions or want to share your experience in caching heavy ASP.NET Core applications, we are waiting for your comments.
