The BIG-IP device by F5 Networks uses an operating system and embedded database management system to stay afloat in a sea of real-time data.

These applications' need for high performance, availability, and reliability--and F5 Networks' desire for scalability and extensibility, so the device's features can keep growing--led the company to design and build a specialized embedded operating system for the BIG-IP product family.

In this article, we describe why we enhanced BIG-IP with an embedded operating system and an embedded data management system, specifically what technical issues we were trying to avoid. This approach could be of use to embedded systems engineers developing networking, communications, or any other type of complex real-time application in which multiple processes read and update shared data in order to perform their work.

An operating system for BIG-IP
The Traffic Management Operating System (TMOS) is the operating system we built for BIG-IP. It's an embedded device platform that is highly optimized for delivering applications over the Internet and other IP networks, and gives the device a performance edge. BIG-IP's popularity has drawn attention to TMOS, and while nearly all the notice has been positive, our operating system has sometimes been mischaracterized as a Linux variant. In fact, BIG-IP does include a copy of Linux, which runs alongside TMOS and provides certain management tasks, such as the command line and Web graphical user interface.

However, the packets flowing through BIG-IP are not "touched" by Linux in any way. Every important system aspect is contained within TMOS and optimized for high-speed, high-volume traffic-management applications. TMOS has its own microkernel, the Traffic Management Microkernel (TMM).

BIG-IP's platform reimplements standard operating system features such as the scheduler and Secure Sockets Layer stack to achieve lower overhead and higher flexibility. It also adds new capabilities, such as iRules, a scripting language (based on Transaction Control Language) for real-time manipulation and modification of traffic, to enable users to achieve complex networking goals.

Rethinking the wheel
In rethinking operating system features in terms of their potential for improving our networking device, we targeted configuration management for a complete reengineering. In Linux and UNIX platforms, provisioning applications and services with settings is handled via a configuration file, usually in ASCII format, that is accessed at startup and during operation.

The problems of this approach are well known. One is a lack of centralization: multiple programs running on a machine generate a profusion of config files that are stored in different places. In these scattered locations, the files are unprotected and sometimes difficult to find. In addition, configuration files' formatting is nonstandard, so that updating them requires knowledge of each file's layout.

In addition to these general problems, our engineers identified requirements for configuration management specific to a multifunction, real-time networking device. One is performance. Network traffic management inherently occurs in real time. With ASCII configuration files, data is saved to disk, making for slow retrieval; often a process must navigate a lengthy configuration file, with multiple disk I/Os, to retrieve or check a single configuration value. During BIG-IP's development, it became clear that only in-memory hosting of configuration data was likely to provide the real-time storage, sorting, and retrieval that the technology requires.

A second demand that drove our software engineers to build a dramatically different configuration engine within TMOS was for large-scale configuration management. At any given time, hundreds of processes may be running inside the BIG-IP device, resulting in the need to manage an unusually high volume of configuration data. A third requirement was support for data sharing between processes.

To a much greater extent than in typical UNIX and Linux boxes, processes running in BIG-IP must "know" about the same data objects, which range from virtual servers, which provide outward-facing IP addresses that redirect traffic to servers and other resources; to trunks, or logical groupings of interfaces; to static routing information used in the device's layer 3 switching functions; to many other object types (we've counted approximately 500 different configuration object types).

A fourth requirement was complex interrelationships between objects. Multiple objects can reference the same real-world entity, such as an IP address; and less complex objects can be combined into more complex ones. Because these objects are shared and interrelated, in the TMOS design it was essential to centralize the objects' location and provide a mechanism for cross-object validation and synchronization, to protect data integrity.

A database system in TMOS
What we also needed was linear performance in BIG-IP's configuration management. If a customer using the device has 100 virtual IP addresses that have to be managed in a certain way, and the time for doing that is X, then the time for doing the same thing to 200 virtual IP addresses has to be two-times-X. What F5 Networks--and ultimately, its customers--can't tolerate is exponential degradation of performance in organizing, querying, and retrieving such data.

In short, to tackle configuration management in BIG-IP, the platform needed to manage a large volume of potentially complex data, as well as data relationships--and so it needed a database management system (DBMS). If you're familiar only with ordinary relational database management systems (RDBMS) used in business or other "non-real-time" applications, a database might seem like the last thing we'd want.

RDBMS technology has a reputation for slowness--at least, compared with the performance demands in embedded, real-time systems like BIG-IP. Traditional database systems are typically hardwired to save records to disk, and this is particularly draining on performance, although such DBMSs do have complex caching logic that keeps some frequently requested data in memory, for faster access.

Our performance requirements, in contrast, demanded a database entirely in memory, without any disk I/O, and preferably without caching (why bother with it when all data is already held in memory?). We found the right fit in an embedded DBMS that was designed from scratch to reside in main memory, as shown in Figure 1.

Its design eliminates not only mechanical disk I/O, but also the file I/O, caching and related logic that would burden a traditional disk-based DBMS's performance even if it were deployed in memory (such as on a RAM disk). Because the embedded DBMS is streamlined, its code footprint and processing demands are quite low--and the extra RAM and CPU cycles this frees up can always be used to benefit the applications running on BIG-IP.

Looking at Figure 1, you can see the key elements of this embedded database system includes a development application programming interface (API) with a static function library for common tasks such as opening and closing the database. Most of the API is generated dynamically as a by-product of compiling the database definition language, and is highly optimized for that definition. Supported indexes include Hash, B-Tree, R-Tree, Patricia trie, and custom. Notably absent from the design are caching and file management layers, which are made redundant in this all-in-memory architecture.

The embedded DBMS was integrated in BIG-IP's control plane, as part of the Master Control Program (MCP). The MCP serves as a collection and distribution point for all configuration data, whether provided by the user or internal to the device itself (Figure 2). The MCP is responsible for validation, or enforcing rules when a user or process attempts to change a data object, and for using BIG-IP's internal messaging to push updated data to processes that subscribe to a subset of the configuration database in order to receive data needed for their operation.