Distributed Systems & Distributed Computing Part I

Nearly all the software we use today is an extent to distributed systems or involved in distributed computing. How? You’re going to know the answer in a bit. For now, let me just give you some examples. Distributed systems or distributed computing is all around us; Google search engine, Amazon platforms, Netflix, Blockchain, online gaming, money transfer, online banking and the list goes on. Probably, the most straightforward and the simplest example of distributed systems is the client-server model which I assume you are familiar with. Let’s go back to the fundamental question and see what exactly a distributed system is.

Distributed system is a collection of separate and independent software or hardware components called nodes that are linked together by means of a network and work together coherently by coordinating and communicating through a message passing or events to fulfill one end goal. Nodes of the system could be unstructured or highly structured depending on the system requirements. In any case, the complexities of the system remain hidden to the end-user be it a human being or a computer and the whole system appeared as a single computer to its users.

So basically, it’s just a bunch of independent computers that cooperate to solve a problem together. I know it sounds simple but it’s a hell of a world under the hood. Before we continue, I just want to say that two programs communicate with each other on the same computer is not necessarily a distributed system. Even though they work together to achieve the same goal, a client-server model that uses the same computer is not a distributed system.

For systems to be called distributed as opposed to centralized or parallel, the following conditions needs to be true.

1. No Shared Clock. Computers have clocks also called timers which are critical electronic devices that keep track of oscillations, and they help the computer to have its own notion of time. It helps to determine the order between events, regulates time and speed of the computer operations. If two programs communicate using the same computer, they basically have the same clock.
2. No Shared Memory. This is another key feature of distributed systems. This means nearly each process has its own independent memory to work with.
3. Concurrency. This is another important characteristic of distributed systems which means that software and hardware components of the system also called processes are autonomous and execute tasks concurrently.
4. Heterogeneity and Loose Coupling. This means the processors are independent, separate from each other and have different speeds.

Before moving on to basic concepts about distributed systems, I just want to mention a couple of notes.

• First, distributed system is a dynamic system that allows computers or nodes to join and leave. This has many advantages but also it introduces some challenges in overhead such as in case of open distributed systems, security issues and the extra work of managing the organization and membership of nodes.

• Secondly, nearly all existing large distributed systems, especially modern ones, are overlaying networks. An overlay network is just a network on top of another network. For instance, Peer-to-Peer networks such as Blockchain and BitTorrent. Voice over IP (VoIP) is another network over the Internet.

I hope by now, you have a fair idea about what a distributed system is and can tell whether system is distributed or not. Now, let’s talk about some important basic concepts about distributed computing.

Distributed Computing is a type of computing over distributed systems which means that distributed computing is more than distributed systems. it’s a broader term, and it is concerned with building and establishing computing models for distributed systems and working out algorithms to solve problems related to such systems. Cloud computing is a good modern example of distributed computing. Other examples of distributed computing are PaaS (Platform as a Service) , IaaS (Infrastructure as a Service), Serverless and etc.

Now, we will keep things simple and talk about basic distributed computing concepts.

One of the very basic concepts that you should know about is the notion of a Node. Node is a software or hardware components that has its own processor and memory. It is able to communicate with the rest of the system. Nodes form open groups, a network of nodes, that is open to the external world. Thus, joining the network is easy and also external entities can communicate with the system easily. The internet is one big giant distributed system that falls under this category. Nodes can also form closed groups which are restricted in terms of membership, authentication, and resource accessibility.

Another important notion is what is referred as a Resource. A resource is an asset in the distributed system that could be accessed remotely by nodes in the network or users of the distributed applications that are over the distributed system. Resources can be virtually anything such as files, services, storage facilities, and etc. Basically, resource is anything that a node can take advantage of or use.

Distribution Transparency is another important concept which denotes that everything that happens under the hood should stay invisible to the end users of the system. In other words, the dispersion of resources , failure of nodes, failovers, migration, replication operations, and etc. all should be invisible to users of the system. The way to achieve this is through an important component called the middleware.

Middleware in this context is a distributed systems layer that connects the nodes together and make them appear as one single supercomputer. It is a logical layer on top of the whole system. Think of it as an operating system that runs over the nodes collectively. Once this layer is established, distributed applications could be built and run over it. This middleware layer manages resources, provides communication & security services, handles failures, and the other complexities of distributed computing.

Furthermore, Concurrency is the fact that multiple operations and activities are executed in parallel. These activities can interact among each other to perform a particular operation. In the diagram below, a simple distributed program is visualized. We could see that in ‘phase 1’, ‘task 1’ and ‘task 2’ are executed in parallel, and there is a simple interaction between them. ‘phase 2’ has three operations in parallel in one cooperation between two of them. As we learned earlier, concurrency is an intrinsic property of distributed systems.

Coordination and Synchronization. These are two important concepts that tackle the problems of no shared clock between processes. Also, they solve the problem of data corruption and inconsistency if two entities try to access data at the same time. Coordination ensure the smooth collaboration between operations and activities and help achieve agreement among them. Synchronization on the other hand, orders and controls access to shared resources.

The next concept is the Architectural Model. The architectural model dictates how the nodes in the system are organized. It defines the structure of the network as well as how nodes communicate and interact. The architectural model is extremely important in distributed computing for the sake of better management of the complexities of such systems and also for ease of maintenance. Mainly, the architecture is needed for software components. That’s because as we said earlier, most distributed systems are overlay networks.

Final concept is Global State. Global state in distributed systems is the union of the states of the separate processes. It is sort of a global view of the system that describes its properties at a particular point in time. Global State is equivalent of a global objects that contains all the global variables used by the software system.