Traditionally, software architects and developers would have expected their applications to run on a powerful server. Recently, however, the trend towards horizontal scaling has been increasing. Rather than expecting applications to run on highly scalable servers, developers havebeen redesigning (or refactoring) Information Systems and software applications so that they can scale horiizontally across a number of computer servers. This refactoring of applications is not a trivial task as both the applications and the data captured, managed and stored by these applications must be designed so that both processing and data can be broken down into smaller chunks. It is this existing architectural trend that has been a key factor propelling the adoption of cloud computing.

There are examples of horizontal scaling in High Performance Computing (HPC), Database Management Systems, CPU-intensive processing and Data-intensive processing.  Horizontal scaling had been used for HPC workloads long before the advent of cloud computing in a Grid Computing framework. Developers have refactored applications to achieve the distribution of HPC workloads across bare-metal compute grids. HPC has been used in many scientific applications. For example, scientists have broken down data for applications such as 3-D climate modelling so that it can be spread across a large number of servers. Grid computing is a predecessor to cloud computing as it uses tools to provision and manage multiple racks of physical servers so that they can all work together to solve a problem. As HPC is extremely demanding in terms of compute power, interprocess communication and input-output (I/O), such workloads would be most suitable for Clouds that provide Infrastructure As A Service (IaaS). Access to bare-metal servers or Type I Virtual Machines (VM) that provide more direct access to I/O devices would be specific examples.

As a sidebar we will define the terms bare-metal and Type I VMs. Bare-metal refers to the underlying physical architecture of a computer or server. Running an Operating System on bare-metal refers to running an unmodified version of the OS on the physical hardware. A Type I or Native VM refers to the scenario where the software layer that provides the virtualization for the VM runs on the bare hardware. Given that many HPC applications leverage the hardware directly for purposes of speed, it is clear that Bare-metal servers or Type I VMs would be suited for such applications.

Database Management Systems can also be adapted to run in cloud computing environments. Database servers can be horizontally scaled and database tables can be partitioned across the servers. This technique is known as sharding and allows multiple instances of database software, be it Oracle, MySQL, SQL Server or any other type of database, to scale performance in a cloud computing environment. Rather than accessing a single, central database, applications now access one of the many database instances depending on which shard contains the requested data.

CPU intensive applications are also good candidates for horizontal scaling. Applications that perform intensive tasks such as frame rendering (the process of transforming logical objects such as points, lines etc. into physical representations) can create a separate VM to render each frame rather than creating a new programming thread, thus enchancing performance. Horizontal scaling is also suitable for data-intensive processing as large amounts of data can be processed and the results coalesced to a coordinating process. For example, Hadoop (http://hadoop.apache.org/), which we discussed in a previous blog, is an open source implementation of the MapReduce framework for processing huge datasets using multiple computers.

The question we will ask in this blog is the role Horizontal Scaling can play for Smart Objects and Smart Networks. As we have illustrated in previous blogs, smart objects provide a rich new pool of real-time or near real-time data that will require processing. This data will need to be processed and stored. Frequently, the data will be need to be converted to meaningful information. For example, applications processing Wireless Sensor Network (WSN) data may have to apply complex mathematical formulae to convert measurements from engineering units to a more meaningful metric. When multiple instances of such data is arriving to a system every second it is clear that such applications may be both CPU and data intensive and would benefit from a horizontal scaling strategy. Similar logic also applies for storing smart object data. It may be difficult to predict the data storage requirements for smart networks at the outset. Rather than running the risk of a single database server becoming full, the data can be categorised and distributed across the cloud.

The use of data from smart networks for intensive tasks such as data mining, prediction formulation and pattern detection/analysis would certainly be CPU and data intensive tasks that would also be candidates for a HPC application. HPC could also be used for simulating and testing smart networks. One example is the use of HPC by the US Army Redstone Technical Test Center to test Ad-Hoc Wireless Sensor Networks. Given the role of smart objects in environmental monitoring and scientific applications - for example, biosensors to detect the presence of chemicals or other agents or monitor human and animal health - one would expect the use of HPC to process smart object data to grow in the coming years.