Skip to main content

Clustering

note

This section does not provide a step-by-step guide for up a cluster. Instead, it offers an overview of the components and requirements for configuring a cluster with Fusion File Share Server. Cluster setup is highly dependent on the specific environment and organizational requirements, varying from one installation to another.

For examples of how to set up clustered Fusion File Share Server installations, refer to the Active-Active Clustering and Active-Passive Clustering howto guides.

Introduction​

Fusion File Share Server supports clustering features that provide Continuous Availability (CA), Scale-Out, and High Availability (HA). The following concepts highlight the benefits of clustering, and complement each other:

  • Continuous Availability:​

    Continuous availability ensures that, in the event of a cluster node failure, another node takes over, providing uninterrupted service to all clients. This feature is critical for SMB servers, where maintaining active connections without disruption is essential. If a node fails, continuous availability mechanisms enable the new node to take over both new client requests and existing SMB connections from the failed node.

  • Scale-Out:​

    Scale-out is a strategy characterized by adding nodes to a system to handle increased load, rather than enhancing the existing nodes' capacity. This distributes workloads across multiple servers, allowing the system to handle more requests and efficiently serve more clients.

    In the context of an SMB server, scale-out enables seamless expansion as demand increases, ensuring high performance and maintaining service reliability.

  • High Availability:​

    High availability combines continuous availability with scale-out, allowing the system to handle increased loads by adding more nodes while maintaining uninterrupted service during node failures. This approach is ideal for environments where both performance and fault tolerance are critical, ensuring scalability without compromising availability.

Based on fault tolerance and performance requirements, you can configure Fusion File Share Server in two modes, Active-Passive and Active-Active:

  • Active-Passive:​

    Active-passive is a clustering configuration for Fusion File Share Server that is the easiest to set up. However, among the three benefits of clustering, it only provides continuous availability. In this mode, only one node is active at a time, with standby node(s) ready to take over in case of failure. This setup requires a failover process, during which existing SMB connections and file operations may hang for a few seconds until the failover is complete.

    The simplicity of this configuration makes it ideal for environments where ease of setup and maintenance are a priority.

    Active-passive clustering is suitable for business-critical applications where the cost of downtime is high, but a single SMB server is sufficient to handle the workload. It provides a cost-effective solution for ensuring continuous availability without complex configuration.

  • Active-Active:​

    Active-active is a clustering configuration for Fusion File Share Server that provides all three benefits of clustering: continuous availability, scale-out, and high availability. In this mode, multiple servers operate simultaneously, forming a cluster where connections and workloads are distributed across all nodes. This approach leverages the combined networking bandwidth of all servers. It ensures that, in the event of a node failure, only a small number of clients are affected and must wait for their connections to resume on another server.

    This configuration is ideal for environments requiring minimal downtime and the ability to scale by adding more servers to handle increased loads, though it is more complex to set up and manage.

    Common use cases for active-active clustering include:

    • Business-Critical Applications in Larger Environments:

      Active-active clustering is often used in large environments where downtime is costly and a single SMB server cannot handle the workload.

    • Environments with High Bandwidth and Low Latency Demands:

      Active-active clustering is commonly deployed where bandwidth demands are high. Examples include:

      • Video Production: Multiple users work on different projects simultaneously, requiring substantial bandwidth.
      • Low-Latency Media Streaming: Large numbers of clients require high bandwidth and cannot tolerate buffering delays.
note

Clustering features typically incur a performance overhead, since they perform caching and persisting the server's state, and coordinating between nodes.

Fusion File Share Server as Part of Your CA/HA Solution​

Fusion File Share Server is an essential component in a continuously available, highly available, and scalable file sharing solution. Its primary role is high-performance handling of the SMB protocol. However, it relies on the rest of your infrastructure to effectively coordinate its behavior with other nodes in the cluster. This infrastructure consists of:

  • Shared Storage:
    One or more file systems accessible by all nodes in the cluster. This storage holds the files to be shared by Fusion File Share Server, along with the server's configuration and persistent state.
  • Properly Configured Networking:
    A network layout and configuration that enables all nodes to communicate with each other and with the clients, including a method to distribute client connections across all nodes in the cluster.
  • Clustering Software:
    Software that manages the communication between the nodes in the cluster, coordinates internal state, and ensures that the cluster behaves as a single entity.

The Fusion File Share Server neither provides nor prescribes specific infrastructure components or software solutions for clustering. Instead, it integrates with the clustering solutions chosen by the customer. It is the customer's responsibility to determine and implement shared storage, networking, and load balancing solutions.

Additionally, ensuring redundancy in underlying services such as storage and network is critical, as the overall reliability of the cluster tends to hinge on its weakest point. While established practices are available, the modular design of the Fusion File Share Server provides flexibility in selecting the components best suited to your specific environment.

Shared Storage and Persistent State​

In both active-passive and active-active configurations, shared storage is accessible by all nodes in the cluster. This storage will be used for:

  1. Data storage–the files to be shared with Fusion File Share Server.
  2. Fusion File Share Server configuration, which includes:
  3. Persistent state and metadata, including information about the ongoing operations and the state of the server and clients:
    • The persistent file handle database.
    • The connection recovery database.
    • The privilege database.
important

For active-passive configurations, all shares' vfs parameters should be set to libc:force_sync.

Mount Points​

When setting up a cluster, configure shared storage mount points and mount options identically on all nodes. Whether using one device or multiple, they should be mounted on the same mount points on all nodes. This consistency is necessary since Fusion File Share Server shares the same configuration files across nodes, where the mount points are defined.

File System​

For active-active configurations, you need a file system that can be mounted read-write on all nodes simultaneously. Such file systems include (but are not limited to):

  • GlusterFS
  • WekaFS
  • GFS2
  • OCFS2
  • Lustre
  • CephFS
  • GPFS

For active-passive configurations, the file system requirements are more relaxed. To reduce failover times, you might consider a distributed or clustered file system, though it is not strictly required. If failover time is not a concern, you can use a non-clustered file system like Microsoft NTFS by Tuxera, ext4, XFS, ZFS, or any other file system that suits your needs.

Storage Redundancy​

While not required by Fusion File Share Server, it is recommended to use redundant storage controllers, network connections, and storage arrays. Configuring drives in RAID and utilizing multipathing can further enhance fault tolerance and availability.

Networking and Load Balancing​

Beyond providing redundant network connections on Multiple NICs connected to different switches, there are additional considerations when setting up networking for your Fusion File Share Server cluster.

Active-Passive and Floating IPs​

In an active-passive configuration, a floating IP address provides a single point of access to the cluster. This IP address is assigned to the active node and, in the event of a failure, is reassigned to the standby node. Clients use this IP address to connect to the server, while the health of the nodes and the failover process are typically managed by clustering software (e.g., Pacemaker).

Active-Active and Load Balancing​

For active-active configurations, each server has its own active IP address that clients can connect to. Typically, rudimentary load balancing can be performed by DNS round-robin, where multiple IP addresses are returned for the same domain name, allowing clients to connect to different servers in the cluster.

While DNS is the most common method for distributing client connections across multiple SMB servers, other name resolution methods can be used, such as editing the hosts file on server and client machines. Regardless of the method, it is essential to ensure that clients can connect to all servers in the cluster and that the servers can communicate with each other.

Network Separation​

It is recommended to separate network traffic between cluster nodes (primarily used by the clustering software) from traffic between clients and servers. This separation can be achieved by using separate network interfaces, VLANs, or physical networks. This ensures that internal cluster communication, which requires the lowest possible latency and collision rate, is unaffected by the client traffic that could saturate the network under heavy load.

Network Redundancy​

As with storage, while not required by Fusion File Share Server, it is recommended to use redundant network connections, switches, and routers to ensure fault tolerance and availability.

Clustering Software​

Clustering software is a generic term for software that manages communication between cluster nodes, coordinates internal state, and ensures that the cluster operates as a single entity. It monitors node health, detects failures, and initiates failover processes.

Fusion File Share Server does not provide clustering software. Instead, it leverages the Corosync Cluster Engine for reliable messaging between nodes and quorum establishment.

For health monitoring and failover coordination, Fusion File Share Server typically relies on Pacemaker, an open-source high-availability cluster resource manager.

Configuring Fusion File Share Server for Clustering​

To provide continuous availability, scale-out, and high-availability capabilities, Fusion File Share Server uses several essential clustering mechanisms:

  • Shared Configuration: Ensures that all nodes in the cluster behave and present themselves as a single SMB server.
  • Persistent File Handle Database: Ensures that clients can resume work with their open files on another node in the event of a failure.
  • Connection Recovery Database: Enables communication with clients to resume immediately after a node failure, without waiting for clients to time out and reconnect.
  • Scale-Out: Allows additional servers to be added to the cluster to handle increased loads.

For Active-Passive configurations, you need to enable and configure the Persistent File Handle Database and the Connection Recovery Database.

For Active-Active configurations, you must enable and configure all these mechanisms.

Prerequisites​

Before configuring Fusion File Share Server for clustering, ensure that the following prerequisites are met:

  • Shared Storage:

    File system(s) accessible by all nodes in the cluster should be configured on all nodes:

    • For active-active configurations, the file system should be mounted read-write on all nodes simultaneously.
    • For active-passive configurations, the file system should be mounted on at least the active node (and optionally on the passive node).

    The considerations for the number of devices, volumes, and mount points are entirely at the customer's discretion, and should align with your storage infrastructure's performance metrics, file system capabilities, and security requirements. As long as these requirements and infrastructure capabilities are met, any of the following configurations could be valid:

    • A single mount point for all shares, including configuration files and persistent state.
    • Separatemount points for configuration files and the persistent state, with individual mount points for each share.
    • Individualmount points for each share, configuration file, and persistent state.
    • Any other configuration that meets your requirements.

  • Networking:

    A network configuration that enables all nodes to communicate with each other and with clients:

    • For active-passive configurations, configure a floating IP address.
    • For active-active configurations, set up a load balancing mechanism, such as DNS round-robin.
    • Optionally, set up a separate network for communication between the nodes. (Recommended)

  • Clustering Software:

    Clustering software (i.e., Corosync and Pacemaker), should be installed and configured on all nodes in the cluster:

    • For active-passive configurations, the clustering software should be configured to monitor nodes and initiate failover processes when necessary by:
      • Moving the floating IP address to the standby node.
      • Mounting shared storage on the standby node, if required.
      • Starting the Fusion File Share Server service on the standby node using the configuration file on the shared storage.
    • For active-active configurations, the clustering software should be configured to:
      • Continuously monitor node health to ensure only nodes that meet storage, network, and Fusion File Share Server service requirements remain part of the cluster.

Shared Configuration​

Place the Fusion File Share Server configuration file on the shared storage and ensure that all nodes in the cluster can access it. This file should be identical on all nodes and contain the configuration for all shares as well as the global configuration parameters.

When Fusion File Share Server is running in a clustered configuration, updates to the configuration should only be made using the tsmb-cfg utility, to ensure that the changes are applied across all nodes.

Scale-Out​

Fusion File Share Server enables the scale-out capability by allowing multiple servers to run Fusion File Share Server software as part of a single installation. Configuring scale-out is straightforward, requiring only activating the scale-out mode:

  • Scale-Out Disabled:

    When scale-out is disabled, Fusion File Share Server behaves as a single active server. This is the appropriate setting for standalone or active-passive configurations.

  • Scale-Out Enabled:

    When scale-out is enabled, Fusion File Share Server behaves as part of a server cluster, suitable for active-active configurations. In this mode, nodes in the cluster maintain a shared FSA state over Corosync to determine the state of the all open files and the order of operations on them.

    note

    Upon startup in scale-out mode, Fusion File Share Server attempts to join the cluster by connecting to the Corosync cluster engine. If Corosync is not installed on the node, Fusion File Share Server will fail to start.

  • Autonomous Mode:

    In autonomous mode, Fusion File Share Server allows multiple servers to run simultaneously in a cluster without sharing the FSA state. This is useful in very large clusters where the overhead of maintaining a shared FSA state would be too high. In this mode, each server maintains its own FSA state, and the servers do not need to establish consensus on order of operations such as opening or acquiring locks on files. However, this can cause issues if multiple clients access the same file on different nodes and one or more clients attempt to write to it. In autonomous mode, because the shared-mode is not synchronized between server nodes, this would be allowed but would likely result in data corruption.

    The autonomous mode is only suitable for specific workloads, such as:

    • Read-heavy workloads, where the majority of operations are read operations.
    • Sharded workloads, where each client has its own share or directory, and the servers do not have to coordinate on the same files.
    • Append-only workloads, where the servers do not have to coordinate on the same files.
    Use autonomous mode with caution!

    Enable autonomous mode only if you are certain that your workload is suitable for it.

    Enabling autonomous mode for an unsuitable workload can lead to data corruption, data loss, or other unexpected behavior, as it does not strictly conform to the behavior clients typically expect from an SMB server.

Enabling Scale-Out​

Use the following parameter to enable scale-out:

Global Parameter Scale-Out

Value Type: string

Value Format: true|false|autonomous:

  • true: Enables scale-out.
  • false: Disables scale-out.
  • autonomous: Enables scale-out, but without synchronizing the FSA state between nodes.

Default Value: true

Persistent File Handle Database​

In both active-passive and active-active configurations, clients must be able to resume work on another node in the event of a failure. This is achieved by storing information about open file handles in shared storage accessible by all active nodes. When a client reconnects to a new node, the node can seamlessly resume the client's work because it has access to the open file handle information.

note

When scale-out is enabled, the file handle database is maintained in-memory on all nodes, making the configuration of the persistent file handle database optional.

Configuring the Persistent File Handle Database​

Use the following parameters to configure the persistent file handle database:

Global Parameter Enable Persistent File Handle Database
Overridden by Per-Share Parameter: ca

Value Type: boolean

Value Format: true|false

  • true: Enables the persistent file handle database globally.
  • false: Disables the persistent file handle database globally.

Default Value: false

Share Parameter Enable Persistent File Handle Database
Overrides Global Parameter: ca

Value Type: boolean

Value Format: true|false

  • true: Enables the persistent file handle database for the share.
  • false: Disables the persistent file handle database for the share.

Default Value: false

important

Setting ca to true, either globally or per-share, is not the equivalent of Enabling CA in Windows Server. The main distinction is that the CA feature in Windows implies synchronous writes of all data to disk. In Fusion File Share Server, to enable this behavior, you must also set vfs to libc:force_sync on all shares, which is required for active-passive configurations.

Global Parameter Persistent File Handle Database Location

Value Type: string

Possibly Overridden by Per-Share Parameter: ca_params

Value Format: <path>

  • <path> is the path on a shared storage where the Fusion File Share Server stores its persistent file handle database. This path must be accessible by all nodes in the Fusion File Share Server to support continuous or high availability. If not overridden on a per-share basis by the optional <path> portion of the share's ca_params parameter, the path of the the persistent file handle database for each share with continuous availability enabled will be <path>/<share_name>, as determined by the share's netname parameter.

Default Value: none.

Examples:

  • /mnt/shared/ca would store the persistent file handle database in /mnt/shared/ca/<share_name> for each share where continuous availability is enabled.
Share Parameter Persistent File Handle Database Location and Configuration

Value Type: string

Possibly Overrides Global Parameter: ca_path
This parameter is only required when Fusion File Share Server is configured as an active-passive cluster.

Value Format: [<path>][,durable]

  • <path> is the path on a shared storage where the Fusion File Share Server stores its persistent file handle database for this share. This path must be accessible by all nodes in the Fusion File Share Server cluster to support continuous or high availability. This value overrides the global ca_path parameter for this share.
  • durable is an optional flag that indicates whether durable handles on this share should be persisted in the persistent file handle database.

Default Value: <ca_path>/<netname>, as determined by the global ca_path parameter and the share's netname parameter.

Examples:

  • /mnt/shared/ca would store the persistent file handle database in /mnt/shared/ca/<share_name> for each share where continuous availability is enabled.

Example​

The following example assumes you have a shared storage volume mounted at /mnt/shared/:

[global]
    ...
    ca = true
    ca_path = /mnt/shared/_ca
    ...
[/global]

[share]
    netname = sh1
    path = /mnt/shared/sh1
    ...
[/share]

[share]
    netname = sh2
    path = /mnt/shared/sh2
    ca = false
    ...
[/share]

[share]
    netname = sh3
    path = /mnt/shared/sh3
    ca_params = path=/mnt/shared/sh3_ca,durable
    ...
[/share]

The above configuration:

  • Sets the global location of the persistent file handle database to /mnt/shared/_ca.
  • Sets the location of the persistent file handle database for share sh1 to /mnt/shared/_ca/sh1, since the global ca parameter is set to true, and the share's ca and ca_params parameters are not set.
  • Does not store the persistent file handle database for share sh2, since the share's ca parameter is set to false.
  • Sets the location of the persistent file handle database for share sh3 to /mnt/shared/sh3_ca, since the share's ca_params has a path=... component which overrides the global ca_path parameter.
  • Also persists durable handles for share sh3, since its ca_params has the flag component.

Connection Recovery and the Connection Recovery Database​

The connection recovery database stores information about ongoing TCP connections between clients and server nodes. This information is used to recover connections in the event of a node failure. The connection recovery database is stored on shared storage, making it accessible to all nodes in the cluster.

When an active node takes over for a failed node, it reads the connection recovery database to identify which connections to resume, using the TCP Tickle ACK technique.

TCP Tickle is a technique used to maintain the state of an idle TCP connection. Fusion File Share Server uses the TCP Tickle ACK technique, which sends an ACK to the client with invalid fields, such as a bogus sequence number. This triggers a set of TCP exchanges, prompting the client to recognize its connection to the failed node as stale connection and reconnect to the newly active node.

Global Parameter Enable Connection Recovery with TCP Tickle

Value Type: boolean

Value Format: true|false

  • true: Enables connection recovery with TCP tickle.
  • false: Disables connection recovery with TCP tickle.

Default Value: false

Global Parameter Connection Recovery Database Location and Configuration

Value Type: string

Value Format: path=<path>

  • <path> is the path to the connection recovery database on the shared storage.

Default Value: none.

Examples:

  • /mnt/shared/cr would store the connection recovery database in the specified directory