Distributed Computing

Tags	Computer scienceNetwork system
Created	@January 20, 2023 6:31 PM
Updated	@June 10, 2023 8:34 PM

Preface

Prerequisites

Learning ethics

Introduction

What is Distributed Computing?

Characteristics.

Transparency.

Global Scheduler.

Design. Interfaces. Open-close principle.

High-performance distributed computing.

Pervasive systems (Supranet)

The model

A misconception is that distributed systems are microservices, indeed the latter is an implementation of the former.

Types of distribution transparency

Access

Location

Relocation

Migration

Replication

Concurrency

Failure.

Pitfalls

Types of distributed systems

Grid computing.

Cloud computing.

Cluster computing.

Distributed information systems

Enterprise Application Integration

Primitives.

ACID properties.

Pervasive systems

Mobile systems

Embedded Devices

Sensor Networks

Why does Distributed Computing matter to you?

If you have performance issues where the process is parallelizable, you have intensive IO operations, or you need a reliable system you must check out distributed computing.

Research

Ecosystem

Standards, jobs, industry, roles, …

Clouds

Hadoop

Docker

Story

Case of study

Google File System (GFS)

Summary

Code

@startuml
!theme crt-amber
class ComputerSystem
class User

package "Overlay networks" <<Cloud>> {
class Node {
   location
   data
   status
   software process
   hardware device
   resources
  +process(message: Message)
  +send(message: Message)
  +receive(message: Message)
}
package Middleware <<Frame>> {
class "Middleware" as MW {
   reliability
   networked environment
   communicate()
   transact()
   security(service)
   account(service)
   compose(service: Service)
}

package "Application 1" <<Node>> {
  class "Computer 1" {
     local operating system
  }
}
package "Application 2" <<Node>> {
  class "Computer 2" {
     local operating system
  }
  class "Computer 3" {
     local operating system
  }
}

package "Application 3" <<Node>> {
  class "Computer 4" {
     local operating system
  }
}
}
}
note left of Middleware: Operating system : Computer :: Middleware :: Distributed system 

class "Distributed system" as DS {
  trade-offs
 sufficiently are suffienclythe  spread across network.
  false assumptions by Peter Deutsch, = [
     the network is reliable, secure, homogenous,
     the topology does not change,
     latency is zero,
     bandwidth is infinite,
     transport cost is zero,
     there is one administrator
  ]
  distribution transparencies based on user needs and performance = 
[
{degree of  grade: Float, type: 'Access'}, 
{degree of  grade: Float, type: 'Location'},
{degree of  grade: Float, type: 'Relocation'},
{degree of  grade: Float, type: 'Migration'},
{degree of  grade: Float, type: 'Replication'},
{degree of  grade: Float, type: 'Replication'},
{degree of  grade: Float, type: 'Concurrency'},
{degree of  grade: Float, type: 'Failure'}
]
  exploit parallelism()
}
note right: A transparent system is Unix-like in which everything is a file

class "Good Distributed system" as GoodDS {
   goals = [supporting resource sharing, being open, being scalable, flexibility]
   scalability dimensions = [size, geographical, administrative] // scalability means that X factor increases the amount of work the system keeps the same performance.
   opposed to a monolithic system i.e. microservices.
   hide(resource across a network)
   support(resource sharing for all kinds of groupware)
   separate policy from mechanism()
   offer(): Components based on Interface Definition Language //openess = completeness+ neutrality+interoperability+composability+etensibility
   scaling up()
   scaling out(technique: Hiding communication latencies and bandwidth restrictions | Partitioning and distribution | Replication )
   span multiple administrative domains()
}

class DistributedInformationSystems {
  stable
  wealth of networked applications
  difficult interoperability
  entrprise-wide information
}

class Transaction {
  types = ["system call", "library procedure", "language"]
  atomic
  consistent
  isolated
  durable
  begin()
  end()
  abort()
  read()
  write()
  call() //follow logical divisions
}

Transaction -->"parallel" Transaction

Database -->"old days, request/reply behavior" TransactionProcessingMonitor

class Database
DistributedInformationSystems <|--"tp monitor" Database
DistributedInformationSystems <|--"exchange information directly" EnterpriseApplicationIntegration
EnterpriseApplicationIntegration -->"coupling" RemoteProcedureCall
EnterpriseApplicationIntegration -->"coupling" RemoteMethodInvocation
EnterpriseApplicationIntegration -->"publish/subscribe" MessageOrientedMiddleware
MessageOrientedMiddleware ..|> RSS


DistributedInformationSystems  *-- Server
Clients -->"distributed transactions" Server

class PervasiveSystems {
  unstable
  users and system are closer
  unobtrusive
  no system administrator.
  small, battery-powered, mobile nodes
  nodes often have a wireless connection
  ad hoc fashion
  a lot of sensors
  a lot of actuators
}

note right: Internet of Things is its marketing name
class UbiquitousComputing {
   transparency is crucial
   continuously present
   unobtrusiveness is crucial
   user has more interaction, and more implicit actions, our system hides interfaces
   context awareness is any relevant information between the user and the system
   autonomy from human intervention
   artificial intelligence in an open environment
}

class MobileSystems {
  use wireless communication
  location change over time
  disruption-tolerant networks: connectivity is not guaranteed
  redundant communication
}
class SensorNetworks {
  ten to hundreds or thousands of small nodes with each other
  limited resources
  power consumption is critical
  operator's site or network datask queries
  askQueries(abstract query)
}

SensorNetworks --|> TinyDB
SensorNetworks --|> SpecialNodes

MobileSystems  --|> GPS
MobileSystems  --|> CarEquipment
MobileSystems  --|> MANET

PervasiveSystems  <|-- UbiquitousComputing
PervasiveSystems  <|-- MobileSystems
PervasiveSystems  <|-- SensorNetworks

PervasiveSystems *-- Mobile
PervasiveSystems *-- EmbeddedDevices

class HighPerfomance {
  stable
}
class ClusterComputing {
  homogeneity
  nodes = [...similar nodes (hardware and OS) closely connected in LAN]
  use(parallel and compute-intensive program)
}
ClusterComputing *-- MasterNode
ClusterComputing *-- ComputeNode
class GridComputing {
  federation of nodes in form the virtual organization = [...different nodes (hardware, OS, and networks)]
  worldwide
}

GridComputing  *-- Applications
Applications  --> CollectiveLayer
CollectiveLayer  --> ConnectivityLayer
CollectiveLayer  --> ResourceLayer
ConnectivityLayer  --> FabricLayer
ResourceLayer  --> FabricLayer

class CloudComputing {
   nodes = [...outsource()]
   vendors = [Google, Amazon, Azure]
   utility computing
   virtualized resources
   outsource(): grid nodes
   provide facilities()
}

CloudComputing -->"Software-as-a-service (Saas)" Applications
Applications -->"Platform-as-a-service (Paas)" Platforms
Platforms -->"Infrastructure-as-service (IaaS)" Infrastructure
Infrastructure --> Hardware



DS <|-- HighPerfomance
DS <|-- DistributedInformationSystems
DS <|-- PervasiveSystems
HighPerfomance <|-- GridComputing
HighPerfomance <|-- ClusterComputing
GridComputing <|-- CloudComputing

note right: Why? Because it is easy for users to access, share, collaborate, and exchange remote resources.

class Communication
class "Synchronous communication" as sc
class "Asynchronous communication" as asc

class "Batch-processing system" as batch
class "Stream system" as stream
class "Parallel work" as parallel
batch --> asc
parallel --> asc


ComputerSystem <|--"autonomous" DS
DS <|-- GoodDS
Communication <|-- sc
Communication <|-- asc
Node <|-- "Computer 1"
Node <|-- "Computer 2"
Node <|-- "Computer 3"
Node <|-- "Computer 4"
DS *-- Node: Nodes operate as a whole the same way, no matter where, when, and how.
User -right--> "single coherent system" DS
Node --> Communication
Communication --> Node
Node --> batch
Node --> stream
Node --> parallel

GridComputing ..|> MOSIX
DS .|> ContentDeliveryNetwork

ComputerSystem <|-- DecentralizedSystem
class DecentralizedSystem{
 resources are necessarily spread aa cross the network.
}
DecentralizedSystem ..|> Blockchain
@enduml

FAQ

Worked examples

Activity 1.1

Concurrent and Parallel Principles

Key concepts

Concurrency

Parallelism

Load Balancing. Load Shedding:

Synchronic operation is one such that the callee blocks the caller until it is done.

An asynchronous operation is one where the callee doesn’t block the caller until it is done. It can be done with a big single-thread finite-state machine that records an interruption or event listener in the underlying platform. The sequential process model is lost. Coroutines. Event Loop.

isochronous

Deadlocks

Reactive systems vs Deliberative systems

Logging and distributed management

Push-Pull vs Subscribe

throughput

Overlay Network

Latency

Patterns of communication. Shared memory vs Message passing.

Idempotent. The property operation whereby can be repeated multiple times without changing the initial application. You always are going to get the same result if nobody changes the state.

Formally, we said the $operation$ to be idempotent if $operation(x,x)=x$ for all $x \in X$ .

For example, suppose the operation consults the money in my bank account you could ask for it a lot of times but always you have the same amount, so it is idempotent. However, transfer $3145 from my bank account is not idempotent because you’re changing the state.

Model	Properties	Example
Multi-threading	Parallelism, blocking system calls	Worker in the browsers
Single-threaded process	No parallelism, blocking system calls	Typical synchronous function
Finite-state machine	Parallelism, nonblocking system calls, Asynchronous operation	Event loop in the browser.

Starvation: This refers to a situation in which a task is prevented from accessing a resource that it needs, due to the actions of other tasks.

Threads and processes.

Case of study. JavaScript.

Event loop.

Stack, microtasks, macrotasks, promises, callbacks, html elements, and custom events, subscribers.

Case of study. Python.

Case of study. Go.

Case of study. C and C++.

Case of study. Java.

https://replit.com/@sanchezcarlosjr/distributed-systems-2023-1

Case of study. Haskell.

We classify programming languages as stateful and stateless to abstract the concurrent and parallel principles, even though people have named the in different ways over the years. Stateful languages are those that encourage us to assign variables, on another hand, stateless languages are those that don’t encourage us to assign variables. JavaScript, Go, and C is stateful languages meanwhile Haskell and Erlang are stateless. Immutability. Mutability.

Semigroups and monoids

Group theory in cryptography

Case of study. Erlang.

Coroutines

Foundations

Timed Input/Output Automaton (TIOA)

Distributed algorithms

Process calculus

Process Algebra

Algebraic Theory of Processes

π-calculus

References

Distributed Algorithms

Pierce, Benjamin (1996-12-21). "Foundational Calculi for Programming Languages". The Computer Science and Engineering Handbook. CRC Press. pp. 2190–2207. ISBN 0-8493-2909-4.

MIT OpenCourseWare. (2023, January 27). MIT OpenCourseWare. Retrieved from https://ocw.mit.edu/courses/6-046j-design-and-analysis-of-algorithms-spring-2015/resources/lecture-20-asynchronous-distributed-algorithms-shortest-paths-spanning-trees

http://profesores.elo.utfsm.cl/~agv//elo330/

Architectures

Architectural styles

Architectural styles have two main views the software and hardware organization such that the style are a set of components connected to each other and how these are configured into a system. A component is a replaceable modular unit with interfaces [OMG, 2004b] where “replaceable” means during that system operation you can change the implementation or interface to another one, but you must consider changing interfaces is a pain in the ass. You connect them by “connector”, a mechanism that mediates communication, coordination, or cooperation among them. Some of them are remote procedure calls, message passing, or streaming data.

Now, we focus on software architecture, that is, the logical organization into software components of which the most important architectural styles are layered architecture, object-based architecture, and resource-centered architecture.

https://www.youtube.com/watch?v=nH4qjmP2KEE&ab_channel=ByteByteGo

Application layering

Object-based and service-oriented architectures

Microservices

@startuml
component Service1 {
    node Machine1
    node Machine2
    node Machine3
    interface interface1
    folder code1
    database service1
}

component Service2 {
    node Machine4
    node Machine5
    node Machine6
    folder code2
    interface interface2
    database service2
}

component Service3 {
    node Machine7
    node Machine8
    node Machine9
    folder code3
    interface interface3
    database service3
}

component SomeFrontOffice<<Mashup>> {
 interface UserInterface
}
note right: For example, a Web app

component AnotherFrontEnd {
 interface UserInterface2
}

note right: For example, a Mobile app or Desktop

cloud CloudVendor

actor        "User"

UserInterface --> interface1
UserInterface --> interface2
UserInterface --> interface3
UserInterface --> CloudVendor

User --> UserInterface
User --> UserInterface2
UserInterface2 --> interface1

@enduml

Resources-based architectures

Middleware organization

System architecture

Decentralized organizations

System design

Algorithms

Case of study

Astrolabe

Robbert Van Renesse, Kenneth P. Birman, and Werner Vogels. 2003. Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst. 21, 2 (May 2003), 164–206. https://doi.org/10.1145/762483.762485

Globule

Globule – Scalable Web application hosting. (2023, February 09). Retrieved from http://www.globule.org

Jade

Jade Site | Java Agent DEvelopment Framework. (2023, February 09). Retrieved from https://jade.tilab.com

Summary

Code

@startmindmap
!theme superhero
* Architectures
** What it is?
*** Software and hardware components, their organization, and connection.
*** Component is a modular unit with an interface replaceable.
**** The interface ought to be replaceable in runtime.
*** Connection is a set of mechanisms that coordinates components. 
** Why?
*** Because it is a universal principle subdividing a complex system into different layers and components.
** Who is a system designer or architect? and what does?
*** She or he chooses, organizes and implements key
**** Protocols
**** Interfaces
**** Services
**** States (e.g. databases)
**** Components
**** Orchestration
** Architectural styles
*** Layered architectures
**** What?
***** Components on a level or layer i call lower-level layers j, that is, j < i
***** It can be async or sync communication
***** Monolith
***** Vertical distribution
**** Why?
***** It is the most popular architecture.
***** Simple to develop.
***** Simple to deploy.
***** Simple testing.
**** Challenges
***** Strong dependency among layers
***** Hard to scale
***** Redeploy the entire 
**** Whether it is?
***** Communication-protocol stacks
****** Standard Protocols with different implementations
***** The Web
***** Application layering
****** Multitiered architectures
****** Simple client-server architecture. 
******* Request-reply behavior
***** Sun Microsystem’s Network File System (NFS)
****** Traditional MVC
******* User-interface level or views
******* Processing level or controllers
******* Models or data level
*** Service-oriented architectures
**** What?
***** Organization into separate, independent entities that encapsulates a service. 
***** independent processes
***** independent threads
**** Why?
***** Loose dependency among services
**** Whether it is?
***** Services
***** Objects
****** OOP approach based on network, that is distributed objects and proxies.
****** Technologies
******* SOAP
******* RPC
******* CORBA
***** Microservices
****** What is it?
******* A lot of small, mutually independent, and encapsulated services.
******** Size does matter!
******* Each of them runs on a separate network and process.
******* Modularization is key
****** Why?
******* Better scaling
******* New technologies
******* Frequent releases
****** Challenges
******* Slower deployment
******* Service discovery
******* Complex configs
****** Technologies
******* Docker
******* Kubernetes
******* Openshift
******* Istio
***** Coarse-grained service composition
****** What is it?
******* A kind of microservice architecture
******** But some services may not belong to the same organization
****** Why?
******* Our organization serves a Service composition based on a cloud provider.
****** Challenges
******* Service harmony
****** Whether it is?
******* Cloud computing
***** Resource-based architectures
****** What is it?
*******  A collection of resources managed by remote components by CRUD operations
****** Whether it is?
******* GraphQL
******** Queries and Mutations
******** Types
******* Representational State Transfer (REST)
******** PUT, POST, GET, DELETE
******** Users identify resources by a single naming scheme
******** Services offer the same interface as the four operations
******** Messages are fully self-described
******** Stateless execution, that is, after an operation, the component forgets the caller.
******** Based on HTTP
*** Publish-subscribe architectures
**** What is it?
***** Strong separation between processing and coordination
***** The system is a collection of autonomous processes.
***** Temporal dimension
***** Processes must be running at the same time.
***** Referential dimension
****** Exists reference between processes such as name or an identifier
**** Whether it is?
***** Topic-based publish-subscribe systems
***** Content-based publish-subscribe systems
***** Event bus systems
***** RSS
***** Mailbox
***** Shared data space
****** Linda (coordination language)
****** Qlib-Server
***** Map-Reducers schemes
****** GFS
****** Hadoop
***** Reactive schemes
****** RXJS
***** Stream schemes
****** Cell phones
** Middleware
*** Goal
**** Achieve the open-close principle
*** Design patterns
**** Wrappers or Adapters
***** What it is?
****** Special component that offers an acceptable interface.
****** It solves the problem of incompatible interfaces.
**** Why?
***** Allow the construction of adaptive software.
***** Whether it is?
****** A broker that centralizes wrappers. O(n) wrappers.
****** A wrapper by each client. O(n^2) wrappers.
**** Interceptors
***** What is it?
****** A software that breaks the flows and executes another software.
***** Whether it is?
****** Proxy JavaScript
** Symmetrically distributed architectures
*** What is?
**** Horizontal distribution
**** Peer is a client and server at the same time, that is, a servant.
*** Structured peer-to-peer systems
**** Attempt to maintain a specific, deterministic overlay network.
**** Distributed hash tables and hypercubes
*** Unstructured peer-to-peer systems
**** Each node maintains an ad hoc list of neighbors.
**** Random graph
***** Flooding
***** Random walks
*** Hierarchically organized peer-to-peer networks
**** Supernodes and leader election problem
**** Whether it is?
***** BitTorrent
** Hybrid system architectures
*** Cloud computing
**** Outsourcing local computing
***** Contrary to on-promise services
**** Service-level agreements
*****  Hardware
*****  Infrastructure (IaaS)
*****  Platform (PaaS)
*****  Function (FaaS)
*****  Application (SaaS)
*** The edge-cloud architecture
**** Internet-of-Things (IoT)
*** Blockchain architecture
**** The principle operation of a blockchain
***** A node broadcasts a transaction
***** A validator collects transactions into a block
***** A single validated block is broadcast of all the nodes
@endmindmap

Worked examples

Assignment 2

Processes

Threads in distributed systems

The most important thread property is allowing blocking system calls but not blocking the entire process in which the thread is running because it makes it much easier to express multiple logical connections at the same time.

Virtualization

Examples are Docker, Virtual Machines, and Emulators.

Docker,OpenVZ/Virtuozzo,and LXC/LXD.

Clients

If you must establish a high degree of distribution transparency on the client side, that is, you should hide the communication latencies, then the most common is to initiate the communication right off the bat, display something quickly to users, and not block the user interface. All together can be done via threads.

Replicated calls from a client

A replicate call is a remote procedure call (RPC) between a node and the replicas, that is, the other nodes in the system that holds a copy of the data or chunk being requested, where a response may or may not be received due to system design or network problems.

Typically, a replicated call refers to a situation where multiple replicas of a server-side application are involved in processing a client's request.

Code

@startuml
title Replicated Calls

actor Client
entity "Proxy, request-level interceptor" as Proxy
database Replica1
database Replica2
database Replica3
note over Replica1, Replica2: A replica is a node that holds a copy of data or a chunk of it.

Client->Proxy: B.request(query,data) // read or write the data
group n replicated calls
Proxy->Replica1: Forward request
Proxy->Replica2: Forward request
Proxy->Replica3: Forward request
end
Replica1->Proxy: Send response
Replica2->Proxy: Send response
Replica2->Proxy: Send response
Proxy->Client: It applies a pipe() to responses, for example, it could send either the first response, or the last response, or combines all responses if you need agreement or chunks from replicas. 
@enduml

Web browser

Servers

Window X

Window X designates a user’s terminal as hosting the server, while the application is known as the client. It makes sense because on one hand there is a limited resource and $n$ applications wanting to access them, so the Windows X server coordinates the clients and resource interruptions.

Code migration

Webhook

Checkpoint and restore

Parallel and concurrent process

Asynchronous and synchronous Programming

Queue system analysis

https://www.youtube.com/watch?v=z611FLLR3_8&ab_channel=GeoffreyMessier

https://www.youtube.com/watch?v=Hda5tMrLJqc&ab_channel=USENIX

https://www.youtube.com/watch?v=oQGreeij-OE&ab_channel=8thLightInc.

Summary

Code

@startmindmap
* Processes
** What is it?
*** A program in execution
** How can we develop cooperative programs?
*** IPC
** Important advantage over threads
*** Processes don't share the data space
** Threads
*** Why?
**** We require hiding blocking operations
**** We must exploit a multicore system
**** Threads offer easier communication to each other and performance than process
*** When the operating system execute them?
**** Context Switching
**** Thread Switching
**** User space
*** Model
**** 1 process: N user-space threads: M kernel space threads: U cores
** Virtualization
*** What?
**** A proxy that mimics the behavior of another system interface
***** Instruction set architecture (ISA)
***** System calls
***** Application programming interface (API)
***** The software environment
*** Why?
**** Portability
***** Isolation
***** You're decoupling hardware and software
***** Reproducible
**** Maintain legacy projects
**** Flexibility to change underlying systems
*** Whether it is?
**** A process virtual machine
***** Runtime system
****** Java
***** Emulation
****** Wine Linux
**** A native virtual machine monitor
**** A hosted virtual machine monitor
**** Containers
***** Unix mechanisms
****** chroot
****** namespaces
****** union file system
****** control groups
***** Docker+Kubernetes
***** Linux containers (LXC)
***** PlanetLab (Vservers)
**** Nix
** Client architecture
*** Networked user interfaces
**** Users interact with remote servers
***** by a specific protocol
***** by an independent protocol
**** Whether it is?
***** X Window System
*** Virtual desktop environment
**** The user has software to access the virtual desktop environment
**** Whether it is?
***** Chrome OS
***** Web Browser
*** Client-side software for distribution transparency
** Server architecture
*** It waits for incoming requests and takes care of them.
*** General design issues
**** Synchronization
*****  Iterative or sync
*****  Concurrent or async
**** Contacting
**** Interrupting
**** State
***** Stateless
***** Stateful
*** Object servers
**** The server provides an interface to invoke local objects
***** Activation policies
***** Mechanism, Object adapter
**** Whether it is?
***** The Ice Runtime system
***** The Apache Web Server
*** Server clusters
**** A set of machines connected through a network
** Code migration
*** An process is moved from one node to another.
*** Reasons
**** Performance
**** Privacy and security
**** Flexibility
*** Models
**** Sender-initiated migration
**** Receiver-initiated migration
*** Migration in heterogeneous systems
**** Process virtual machine

@endmindmap

Exercises

Assignment 3

Communication

Protocols

Routing algorithms

Deadlock-free packet switching

Wave and Traversal Algorithms

Election algorithms

Termination detection

Anonymous Networks

Snapshots

Sense of direction and orientation

Synchrony in networks

Remote procedure call (RPC)

Webhook

A remote procedure call is a transparent function or method call that internally sends and receives data from remote processes. It must hide the message-passing nature of network communication. You can find a lot of RPC examples such as REST, SOAP, gRPC, Ajax in Web Browsers, graphQL, CORBA, and Trihft.

Message-oriented communication

Multicast communication

Worked examples

Assignment 4

Summary

@startmindmap
!theme materia
* Communication
** OSI Model
*** What is it?
**** Communication protocols
***** Application, Presentation, Session, Transport, Network, Data link, Physical
***** Message = payload + header for each protocol
**** Middleware protocols
***** What is it?
****** General-purpose protocols that warrant their own layer
***** Examples
****** DNS
****** Authentication and authorization protocols
*** Why?
**** It allows thinking about open systems to communicate with each other
*** Examples
**** Connectionless services
***** Email
**** Connection-oriented communication
***** Telephone
** Remote procedure call
*** Why?
**** Achieve transparency
*** What is it?
**** The real implementation is executed in another node over a network
*** Examples
**** API REST
**** GraphQL
**** gRPC
**** CORBA
**** Ajax in browsers
*** How?
**** Local stub communicates with the server via passing messages
**** Transport Procedure Call
**** Transparent calls
**** Marshaling and Serialization
**** Interface Definition Language (IDL)
***** Issue
****** Code migration and references
** Message-oriented communication
*** Issues
**** Does the communication persist?
**** Is the communication synchronous?
***  How?
**** Async communication
**** Sync communication
**** Technologies
***** Socket interfaces
***** Blocking calls
*** Solution
****  Design Patterns
***** Pipeline pattern
***** Publish-subscribe pattern
***** Request-reply pattern
***** Message-queuing systems
***** Message brokers
**** Message-oriented middleware
***** When?
******  When the RPCs are not appropriate.
***** Examples
****** MQTT
****** ZeroMQ
****** The Message-Passing Interface (MPI)
** Multicast comunication
*** What is it?
**** Disseminate information across multiple subscribers.
**** Examples
***** IPFS
*** How?
**** Peer-to-peer system
***** Structured overlay network
***** Every peer sets up a tree
***** Distributed hash table (DHT), Distributed data
**** Flooding messages across the network
**** Epidemic protocols
***** Simple
***** Robust
@endmindmap

Naming

Flat naming

Structured naming

Attribute-based naming

Worked examples

Assignment 5

Summary

Code

mindmap
  root((Naming))
    (Names)
      What?
         It identifies entities
      Whether it is?
          Processes
          Users
          Network connections
      How?
         Naming system
         ::icon(fa fa-ear-listen)
         Access point or address
         Location independent
         ::icon(fa fa-location-dot)
    (Flat naming)
      What?
         Names are random bit strings
         Good for machines
      How?
        Broadcasting
        Multicasting
        Forwarding pointers
        (Home-based approaches)
        Distributed hash tables
        Exploting network proximity
        Hierachical approaches
        Secure flat naming
    (Structured naming)
      What?
        Good for humans
        Naming systems
      How?
        Name spaces
        ::icon(fa fa-diagram-project)
        Name resolution
        Closure mechanism
        Linking and mounting
        Name space distribution
      Whether it is?
        DNS
    (Attribute-based naming)
      What?
        Search based on a description
        ("(Attribute, value)")
       How?
        Directory services
        Decentralzied implementations
        (Space-filling curves)
       Whether it is?
        Resource Description Framework
        Lightweight Directory Acess Protocol
      Mermaid
    (Named-data networking)
      What?
        (Information-centric networking)
        Get a copy to access it locally 
      How?
        New routing

Coordination

Clock synchronization

Summary

Code

mindmap
  root((Coordination))
    [Clock synchronization]
      What?
        It's doing the right thing at the right time
      Why?
        Data synchronization
        Process synchronization
        Time is so basic to the way people think
      Examples
        File timestamps
        Financial brokerage
        Security auditing
        Collaborative sensing
        Real time systems
    [Physical clocks]
      What?
        A computer timer made by machined quartz crystal
        Counter, Holding and Clock Ticks
        CMOS RAM
      Issues
        Leap second
        Clock skew
      Examples
        Coordinated Universal Time
        Exchaning clock values
    [Logical clocks]
      What?
         It's unnecessary knowing the absolute time, but
         Related events at different procceses happen in the correct order
      How?
        Each event e, is assigned an unique timestamp C e
      Examples
         Lamport clocks
         Vector tunestano
    [Mutual exclusion]
      What?
        At most one process at a time has access to a shared resource.
      How?
        It can be achieved easily with a coordinator.
        A fully distributed algorithms exist, but it can go wrong.
    [Election algorithms]
      What?
        Election algorithm choose a new leader in a DS.
      Why?
        If the coordinator is not fixed, it is necessary choose a coordinator.
        When the coordinator crash, you must choose a new leader.
    [Gossip-based coordination]
      What?
        Coordinate a peer to peer system.
      How?
        Choose another peer randomly from an entire overlay.
        Partial view is refreshed regularly and randomly. 
      Examples
        Routing
        Tribler
    [Distributed event matching]
      What?
         To route notifications to proper subscribers.
      How?
         One to one comparisons by topic
         Content based matching
      Examples 
        Publish subscribe systems
    [Location systems]
      What?
        Take proximity into account
      Examples
        Global Positioning System
        Logical positioning of nodes
        Network coordinate system
        Wigle database
        Centralized positioning
        Decentralized positioning

Questions

Assignment 6

Consistency and replication

Stateless and stateful

Atomic objects

Fault tolerance

Fault Tolerance in Distributed systems

Case of study

VMware FT

Circuit Breaker Design Pattern

Questions

Assignment 7

Fault tolerance

Fault tolerance in Asynchronous systems

Fault tolerance in Synchronous systems

Failure detection

Stabilization

Process resilience

Reliable client-server communication

Distributed commit

CAP theorem

Summary

Assignment 8

Security

📯Intelligence

Tutorials and assignments

We will explore some alternatives to perform distributed computing, both locally and in a network, starting from the most basic approaches and progressing to the use of the most feature-rich libraries.

The “&” operator

(sleep 1 && echo "abc") & (sleep 2 && echo "def") & (echo "efg")

GNU parallel

In my opinion, GNU Parallel works well for complex tasks when run locally, but it can be difficult to use on a cluster. You can see for yourself using https://www.msi.umn.edu/support/faq/how-can-i-use-gnu-parallel-run-lot-commands-parallel.

Examples

cd /tmp/ && touch {1..5}.my_random_file | find . -type f -name '*.my_random_file' | parallel gzip --best

parallel echo ::: 😀 😃 😄 😁 😆 😅 😂 🤣 🥲 done!

touch images.txt && parallel printf 'https://picsum.photos/200/300\\n%.0s' ::: {1..1000} | parallel -j+0 -k "echo {} | tee -a images.txt" && cat images.txt | parallel wget

touch images.txt && parallel -j+0 "printf 'https://picsum.photos/200/300\\n%.0s'" ::: {1..2} | parallel -j+0 "echo {} >> images.txt" && parallel -a images.txt -j+0 wget

xargs

cat images.txt | xargs -n 1 -P 10 curl -s | gawk '{print $1}'

Ray

On the other hand, Ray performs well for complex tasks on an on-premise cluster, but when used locally, it can be overwhelming.

https://www.ray.io/

A distributed Hello World program

Network File System

Hadoop or Spark

Open MPI

Low-level alternative

LibP2P

Location-addressing approach vs content addressing

Content identifier

Peer. A participant in a decentralized network that is equally privileged.

Queue messaging

https://mqtt.org/

MQTT

Message broker

Quality of service

OASIS (organization)

ZeroMQ

https://zeromq.org/

Quality of service

Message-oriented middleware

Message queuing service

Your own map-reducer

https://pdos.csail.mit.edu/6.824/labs/lab-mr.html

Other Alternatives

Object storage

1. Network Attached Storage (NAS) - a file-level storage architecture that allows multiple users to access shared files over a network. 2. Distributed File System (DFS) - a protocol that allows users to access files on multiple servers as if they were on a single shared drive. 3. iSCSI (Internet Small Computer System Interface) - a protocol used for connecting storage devices over a network as if they were directly attached to a computer. 4. Remote Direct Memory Access (RDMA) - a protocol used for direct memory access between servers over a network, often used for high-performance computing and data-intensive applications. 6. AFS (Andrew File System) - a distributed file system designed for large-scale collaboration and data sharing built by Carnegie Mellon University. 8. GlusterFS - a distributed file system designed for scalability and high availability, often used for cloud computing and big data applications. 9. Lustre - a parallel distributed file system designed for high-performance computing and scientific applications.

10. File Transfer Protocol (FTP) and Secure File Transfer Protocol (SFTP) - a protocol used for transferring files over a network, often used for website hosting.

version: '3'

services:
  sftp:
    image: atmoz/sftp
    ports:
      - "2222:22"
    environment:
      SFTP_USERS: sftpuser:pass:1001:/data
    volumes:
      - /path/to/sftpdata:/data

11. WebNative FileSystem (WNFS)

MinIO - an open-source, self-hosted cloud-based object storage service that can be deployed on-premises or in a cloud environment.

Ceph - an open-source, software-defined storage system that can be used to create a distributed object store.

OpenStack Swift - an open-source object storage system that can be used to create a scalable and highly available object-store.

Apache Cassandra - a distributed NoSQL database system that can be used to store and retrieve large amounts of data.

GlusterFS - an open-source, distributed file system that can be used to create a scalable and highly available object store.

Riak KV - an open-source, distributed NoSQL database system that can be used to store and retrieve large amounts of data.

OpenIO - an open source, object storage, and data management platform that can be used to store and retrieve large amounts of data.

Scality Ring - an open-source, distributed storage system that can be used to create a highly available and scalable object store.

SwiftStack - an open-source, software-defined storage system that can be used to create a distributed object store.

Nextcloud Object Storage - an open-source, self-hosted cloud storage system that includes an object storage component that can be used to store and retrieve large amounts of data.

SSH and RSYNC over SSH

https://hub.docker.com/r/linuxserver/openssh-server

---
version: "2.1"
services:
  openssh-server:
    image: lscr.io/linuxserver/openssh-server:latest
    container_name: openssh-server
    hostname: openssh-server #optional
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
    volumes:
      - ./config:/config
    ports:
      - 2222:2222
    restart: unless-stopped

Network File System

Server Message Block (SMB) and CIFS/SMB

BitTorrent Sync

Atom (web standard)

Content Management Interoperability Services

Linked Data Platform

Unison File Synchronizer https://github.com/bcpierce00/unison

netcat (nc)

nc -lk 1234 < ~/shared

Simulations

Simgrid

OverSim

Next steps

References

Tanenbaum, Andrew S.; Steen, Maarten van (2002). Distributed systems: principles and paradigms. Upper Saddle River, NJ: Pearson Prentice Hall. ISBN 0-13-088893-1. Archived from the original on 2020-08-12. Retrieved 2020-08-28.

6.5840 Home Page: Spring 2023. (2023, January 19). Retrieved from https://pdos.csail.mit.edu/6.824

"Distributed Programs". Texts in Computer Science. London: Springer London. 2010. pp. 373–406. doi:10.1007/978-1-84882-745-5_11. ISBN 978-1-84882-744-8. ISSN 1868-0941.

Lynch, Nancy. Distributed Algorithms. Burlington, MA: Morgan Kaufmann, 1996. ISBN:9781558603486.

Attiya, Hagit, and Jennifer Welch. Distributed Computing: Fundamentals, Simulations, and Advanced Topics. 2nd ed. New York, NY: Wiley-Interscience, 2004. ISBN: 9780471453246.

Herlihy, Maurice, and Nir Shavit. The Art of Multiprocessor Programming. Burlington, MA: Morgan Kaufmann, 2008. ISBN: 9780123705914.

Guerraoui, Rachid, and Michal Kapalka. Transactional Memory: The Theory. San Rafael, CA: Morgan and Claypool, 2010. ISBN: 9781608450114.

Dolev, Shlomi. Self-Stabilization. Cambridge, MA: MIT Press, 2000. ISBN:9780262041782.

Kaynar, Disun, Nancy Lynch, Roberto Segala, and Frits Vaandrager. The Theory of Timed I/O Automata. 2nd ed. San Rafael, CA: Morgan and Claypool, 2010. ISBN:9781608450022. https://groups.csail.mit.edu/tds/papers/Lynch/Monograph-second-edition.pdf

MIT 6.033

MIT 6.5840

MIT 6.824. (2023, January 27). MIT OpenCourseWare. Retrieved from https://ocw.mit.edu/courses/6-824-distributed-computer-systems-engineering-spring-2006

Systems, Mit 6. 824: Distributed. "Lecture 1: Introduction." YouTube, 6 Feb. 2020, www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB&ab_channel=MIT6.824%3ADistributedSystems.

Saltzer and Kaashoek's Principles of Computer System Design: An Introduction (Morgan Kaufmann 2009).

Nancy Lynch, Michael Merritt, William Weihl, and Alan Fekete. Atomic Transactions. Morgan Kaufmann Publishers, 1994.

MIT OpenCourseWare. (2023, January 27). MIT OpenCourseWare. Retrieved from https://ocw.mit.edu/courses/6-852j-distributed-algorithms-fall-2009

cs6234-16-pds https://www.comp.nus.edu.sg/~rahul/allfiles/cs6234-16-pds.pdf

Stevens, W. Richard, Bill Fenner, and Andrew M. Rudoff. UNIX Network Programming, Vol. 1: The Sockets Networking API. 3rd ed. Reading, MA: Addison-Wesley Professional, 2003. ISBN: 9780131411555.

McKusick, Marshall Kirk, Keith Bostic, Michael J. Karels, and John S. Quarterman. The Design and Implementation of the 4.4 BSD Operating System. Reading, MA: Addison-Wesley Professional, 1996. ISBN: 9780201549799.

Stevens, W. Richard, and Stephen Rago. Advanced Programming in the UNIX Environment. 2nd ed. Reading, MA: Addison-Wesley Professional, 2005. ISBN: 9780201433074.

. "Distributed Systems lecture series." YouTube, 31 Jan. 2023, www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB.

https://ki.pwr.edu.pl/lemiesz/info/Tel.pdf

http://bafybeiemcaemivgxkzkuqq2kvvfmyo32joipgp6pivk2ea32la2stnmkwm.ipfs.localhost:8080/?filename=Maarten van Steen%2C Andrew S. Tanenbaum - Distributed Systems_ Principles and Paradigms-Maarten van Steen (2023).pdf

https://www.youtube.com/@DistributedSystems

https://www.youtube.com/@lindseykuperwithasharpie

https://www.youtube.com/@rbdiwate626