Tracker Ten In Memory Databases

In Memory Databases

Just as the name implies, in memory databases work with information stored directly in your computer’s RAM memory. RAM memory is your computers main memory, not supplementary long-term storage, like your hard drive. In memory databases are best suited for smaller data sets, as RAM is much more expensive than hard disk space. But while the memory is more expensive, it’s also much faster, speeding up data processing times. In fact, you might notice phenomenal speed gains, if you migrate to an in-memory database.

If you are looking for an in-memory database system, you don’t necessarily need to purchase a specific database system. In fact, any type of database technology is likely to have in memory versions including relational databases, documents databases and graph databases. Large scale database system like SQL Server, PostgreSQL and MongoDB can also be configured to store data in memory instead of on disk. However, these special configuration features are sometimes only available in certain database versions. For example, MongoDB only supports in memory storage in its enterprise edition. There might also be open-source extension engines that make a traditional database system work like an in-memory system, like Percona for MongoDB.

Regardless of the in-memory database you select, you need to be aware that your computer’s RAM memory is volatile, meaning that information is only remembered while your computer is on. The minute you turn your computer off, all information stored in RAM is wiped. Therefore, if you are using an in-memory database, you will need to save your data to a drive when you finish, and you will need to reload it when you start. Please note you can get the benefits of fast memory and non-volatility using non-volatile ram (NVRAM). But this type of flash memory is expensive, and there are limitations to the number of times the memory can be erased and overwritten.

In memory databases typically support automatic loading and saving of data, so this usually not a big issue. As an example, SQLite and Tracker Ten are able to load and save data into a single file, with no extra configuration. This is no different then many other applications on your computer, like word processors.

An in-memory database may be as simple as a Redis k or Tarantool key value store, or as complicated as a full-blown set of relational tables with indexes. Either way an in-memory database can be amazingly fast – a comparison between in memory Redis and regular MySQL, shows that Redis is much quicker.

When is an In-Memory Database Used?

In-memory databases can be used for any type of application (our own Tracker Ten system offers in-memory databases with numerous templates). However, there are some applications that in-memory databases are especially suited for. These applications include:

Basically, any application that requires fast processing and multi-user concurrency is a strong contender for an-memory database. This includes almost all OLTP (online transactional processing) applications.

Disadvantages of In-Memory Databases

In-Memory databases do have some disadvantages. As discussed above In-Memory databases typically rely on RAM which is the most expensive type of memory, and its volatile meaning data needs to be saved and loaded each time your device loses power. In fact, when you do need to load and save data (or just back it up from memory) you will likely experience latency issues and memory use surges and spikes, as these operations take a lot of resources. For the above reasons, an in-memory database will not work well for any application that requires the storage of huge amounts of data (like a photo or video database).

Also, in-memory databases are typically not optimized for modern multi-core processors. This means that adding increases cores to your system won’t always significantly speed up your database application, unless you manually implement more complex data storage mechanisms like data shading.

Implementing an In-Memory Database

There are several ways to implement an in-memory OLTP database. First, you may use a modern programming language like Java, C#, Python, php or Ruby. All of these languages will let you define data structures that can be used as a foundation for an in-memory database, and can be programmed to provide needed functions like searching and querying, reporting, and loading and saving of data.

You can also use traditional databases like SQL Server, MySQL and ORACLE with some configuration. Finally, there are off the shelf NOSQL tools like Redis, Apache Ignite, Amazon Elasticache and Tarantool that use in-memory database technology to provide crash resistant interactive data processing. Some of these systems allow multi-tier storage where data is moved automatically between memory and disk, and they allow you to set the amount of RAM used. This multi-tier architecture is similar to caching.

Please note some of these technologies, like Apache Ignite for Redis and in-memory columnstore in Azure work on cloud-based systems. But even though your data may be stored on a remote cloud-based server, all of the same principles apply – data stored in-memory in a remote servers volatile RAM will be much faster than data stored on a remote server’s non-volatile hard drive. And price-tiers for in-memory cloud technology will be significantly higher than traditional cloud-based databases. Specifically, online database pricing is often based on DTU’s – database transaction units. The cost for these DTU’s is significantly higher for in-memory databases, than disk-based databases.

In Memory Database vs Database Caching

To speed performance some database systems like Oracle and some off the shelf database tools like Memcached support in memory caching. This means that commonly accessed data is store in memory, while data that is not accessed often remains on disk. This setup reduces the number of times slower memory like hard drives need to be accessed and allows RAM memory to be used efficiently. However, a lot more programming is required in the database to support intelligent caching, as information needs to be moved automatically between RAM storage and Disk storage.

In Memory Data Grids

An in-memory data grid is similar to an in-memory database, expect that data in a memory data grid may be stored on multiple servers, where an in-memory database is stored on a single system. In fact, an in-memory data grid can scale to thousands of servers. Operationally an in-memory data grid allows RAM to be shared in a clustered network. This type of grid system works by installing special software on every computer on the network, that allows all computers on the network to access RAM memory on all other computers on the network. This architecture allows memory storage space to be distributed (greatly increasing capacity), while maintaining the speed benefits of an in-memory system.

Tracker Ten Desktop Database Application

If you are looking for a fast single user in memory desktop windows database application, please be sure to have a look at our Tracker Ten database system. Tracker Ten has numerous available templates for all types of data tracking applications, and manages all of the complexities of an in-memory database automatically.