How cluster updates run

How a coordinated update works

The primary node decides what to install and when. It publishes a plan, and each participating node installs the update at that time from its own schedule. No node waits for another, so one slow node does not delay the others.

Each node checks with the primary node every time. A node does not rely on the last instruction it received. A node that was turned off for a week gets the current plan when it returns, not the plan it last saw.

Changing a setting replaces the plan. If you change the maintenance window on the primary node, the primary node publishes a new coordinated update and retires the previous one. A node cannot change a coordinated install time by recalculating its own schedule.

When the other nodes receive a scheduled update

When you click Install Later on the primary node, the primary node publishes the plan to the other nodes immediately.

If publishing does not finish, the page shows the following message: The other nodes have not been told about this schedule yet. They will be told at the next automatic update check. This is a delay, not a failure. The schedule is saved on the primary node.

The other nodes receive the plan during the next automatic update check, which runs every eight hours. If a coordinated update is already published, a background check that runs every 15 minutes may deliver the plan sooner.

You cannot retry publishing manually. To confirm that the other nodes received the plan, check the Current panel in the Cluster Updates section on the primary node. The update appears there after it reaches the other nodes.

Patches

Automatic patches, including critical security patches, are not coordinated. Each node installs them on its own schedule, so no node waits for another to receive a security patch.

If you install or schedule a patch from the primary node, that patch is coordinated like any other update.

How a node that falls behind catches up

A participating node can miss a coordinated update if it was turned off, could not be reached, or failed to install the update. The node catches up on its own. It compares its version with the primary node and installs the next update toward that version. A node that was turned off starts catching up when it restarts.

Catching up works as follows:

• One step at a time. A node that is far behind installs interim builds over several checks instead of moving directly to the version of the primary node. A node that was off for a long time can take several hours to catch up. This is expected.
• Never past the primary node. A node does not install a build that is newer than the primary node, or any build if it cannot determine the version of the primary node. A node that is ahead of the primary node cannot be brought back automatically. The Cluster table flags this case.
• Requires the primary node and the update service. If either one cannot be reached, the node skips the attempt and tries again later.
• Slows down after repeated failures. After a step fails three times, the node tries again only during the three daily checks instead of every 15 minutes. A successful install returns the node to the normal schedule.

Patches are also caught up, one at a time, usually during the three daily checks.

📘

A node that does not participate does not catch up. It manages its own updates. After an upgrade to Base Software 8.4, no node on an existing on-premises cluster participates until you turn participation back on, so the nodes can remain on different versions.

When you are notified

Catching up happens without notifications while it works. You are notified in the following cases:

• A node remains behind for four hours. The primary node sends the A Node Is Behind The Other Nodes notification and shows a banner.
• A node has no build that moves it toward the primary node. The node reports that it is behind with no eligible build instead of retrying.

When automatic product upgrades are held

Automatic product upgrades do not install, even when automatic updates are turned on, in the following cases:

• The appliance is in an on-premises failover pair, and the failover backup has Enable Automatic Failover turned on. An update that restarts one appliance could cause the other to take over during the upgrade. Both appliances in the pair show a message on their Updates page.
• The appliance follows a primary node that is in a failover pair with automatic failover turned on. Every node must run the same version, so the nodes behind that primary node also hold automatic product upgrades. These nodes show the message Automatic product upgrades are paused on this node and name the primary node.

These holds apply whether or not the node participates in cluster updates. A node that does not participate still has automatic product upgrades held if it is in, or follows, a failover pair with automatic failover turned on.

When the failover state is unknown

If the failover pair has not reported its state within the last hour, or the last attempt to reach the primary node failed, the appliance treats automatic failover as turned on and holds automatic product upgrades. A connection problem can therefore look like upgrades have stopped.

Read the message on the Updates page carefully. It distinguishes between automatic failover that is turned on and automatic failover that has not reported whether it is turned on.

What is not held

• Manual product upgrades. You can still click Install Now or Install Later on the primary node. The update is coordinated and installs on every participating node. The Updates page shows a warning that the other appliance in the pair might take over while one restarts.
  To prevent the takeover, turn off Enable Automatic Failover on the failover backup before you start the upgrade, and turn it back on after both appliances run the same version.
• Patches, including critical patches. These continue to install on each node on its own schedule.

When the primary role moves to another appliance

After a failover swap, or when another node becomes the primary node of a cluster, a scheduled update does not move with the primary role. Within 15 minutes, the previous primary node withdraws the update. If the install time has not passed, it sends the Cluster Update: Withdrawn notification. An install that is already running finishes first.

To install the update:

• If the update was scheduled automatically, the new primary node schedules it again at its next automatic update check, as long as Automatically install the latest updates during maintenance day(s) and time is turned on and automatic product upgrades are not held. The new primary node's minimum days notice applies, so the update can install a week or more later than planned.
• If you scheduled the update, schedule it again on the new primary node.

If the previous primary node does not participate in cluster updates, it keeps the update on its own schedule and can end up on a newer version than the new primary node. This is typical on a failover pair that has just upgraded to Base Software 8.4. Failover operations stop until both appliances run the same version.

If the previous primary node is down, it cannot send the notification. After a failover swap, check the Updates page on the new primary node.

If you disband the cluster or break the failover relationship, the update is withdrawn the same way. The appliance that was the primary node still installs the update at the scheduled time.