A Load Balancing Backup Story

Let's continue where we stopped last time...

NetScaler Spillover Feature

The scenario from Part 1 is not really satisfying our condition for Backup Servers per datacenter. Let’s examine a feature that might just be the right tool.

The spillover feature is designed to control the Backup Virtual Server steering based on certain condition or in another words when should the Virtual Backup Server kick in. By default (without a configured spillover method) this will happen when the Primary Virtual Server is marked DOWN (all 4 Servers in our case should be unreachable). With spillover a threshold should be configured and once reached it will perform spillover (Failover to the Backup Virtual Server). The backup virtual server manages the connections it receives until the load on the primary falls below the threshold. There are two types of spillover methods - Predefined and Policy Based. A spillover configuration on the appliance consists of the following config pieces:

• Primary Virtual Server with a spillover method
• Predefined spillover methods
• CONNECTION - Occurs when the number of client connections exceeds the specified threshold in the spillover settings.
• DYNAMICCONNECTION - Occurs when the number of client connections at the virtual server exceeds the sum of the Max Clients settings of the bound services.
• BANDWIDTH - Occurs when the bandwidth consumed by the virtual server's incoming and outgoing traffic exceeds the threshold
• HEALTH - Occurs as the percentage of the services that are UP drops below the threshold. (Service weights should be calculated)
• Policy Based Spillover
• Expression Based Condition for spillover to occur (Custom defined by you)
• A threshold (for the spillover method - condition)
• Each spillover method should have a configured threshold
• A Backup Virtual Server (a spillover target for the traffic steering)
• Virtual Load Balancer containing the Backup Virtual Services (Servers)

For the majority of the setups the predefined spillover methods will do just fine. The idea is to chain the Load Balancer Services one after another. (There is a Redirect URL in the spillover settings intended for the case when all the Virtual Servers in the chain becomes unavailable to redirect you to a Informational\Maintenance Page. This must be configured on the first Virtual Server in the chain. People widely using it for HTTP to HTTPS redirection.)

The Policy Based Spillover gives you the flexibility for combining the predefined methods as well as a lot more expressions-based conditions like Response Time, State (UP, DOWN, OUT OF SERVICE), Active Transactions, Active Services, Auto Scale Members, SNMP Load Values, etc. I’m not ware of any chaining limit, but you should remember that Persistence and Timeout settings apply. If you configure spillover persistence, the backup virtual server continues to process the connections it received, even after the load on the primary falls below the threshold. Same logic applies on the timeout. Nevertheless, there is no out of the box solution when specific half part of the Servers is overloaded\unavailable. The HEALTH percentage spillover method (say threshold 50%) is general for the whole Virtual Load Balancer (any two DOWN). This doesn’t work for us. We want the Backup to kick in when two of a Site are DOWN.

But we can check the STATE of individual Backend Servers via Policy Based Spillover. We can do it with policy expressions. Unfortunately, we have to specify each member of the service group (both Servers), not directly to reference the Service Group itself. The expression is below: (Citrix uses logical operators like “&&”, “||”, “==” etc. more info here )

(SYS.SERVICEGROUP("LB_SG_RGBAPP_PRIMARY_S1").MEMBER("169.254.10.99",80).STATE.NE(UP)&&SYS.SERVICEGROUP("LB_SG_RGBAPP_PRIMARY_S1").MEMBER("169.254.10.100",80).STATE.NE(UP))||(SYS.SERVICEGROUP("LB_SG_RGBAPP_PRIMARY_S2").MEMBER("169.254.10.101",80).STATE.NE(UP)&&SYS.SERVICEGROUP("LB_SG_RGBAPP_PRIMARY_S2").MEMBER("169.254.10.102",80).STATE.NE(UP))

The expression will check if the members (4) of the two PRIMARY Server Groups at Site 1 & Site 2 have different STATE than “UP”. Policy Based Spillover rely on the expression returning “TRUE”. We use negative expression, because we want the policy to evaluate to “TRUE” and to be triggered even when we manually place the Servers in “OUT OF SERVICE” (most likely due to Maintenance reasons). We combine the member servers’ STATE so we can evaluate the Service Group STATE as a whole. As a result, we’ll know if a Service Group has STATE different than “UP”!

At this point we’ll need to establish at least some naming convention, otherwise it will be difficult to follow:

S1 - Site 1 PRI - PRIMARY Servers S2 - Site 2 BCK - BACKUP Servers

Example: S1_PRI - Site 1 PRIMARY Server Group - (containing Server #1 & Server #2)

Now let’s chain the servers that we have. There are not that many combinations of Primary and Backup Server Groups (having two Sites). We have 4 Server Groups that equals 6 pairs of possible combinations. We won’t deal with the individual servers for now and the requirement for minimum of two active Servers, we’ll get back to that later. Let’s create 6 Virtual Load Balancers and chain them. The diagram below illustrates the chain logic or the tiers we have. Each tier must represent one of the possible combinations.

Each tier will need 4 Servers divided into two Server Groups which STATE we’ll monitor with the Spillover policy. The Server Groups per Tier are represented by the below table where you can simulate DOWN events of certain Service Groups. The first tier not affected by DOWN event (red coloring) will handle the traffic load. Yellow coloring indicates individual servers going down, that should not trigger spillover (based on our expressions).

I would recommend using such table whenever you configure the spillover feature.

You might have noticed that the expressions in the above table are slightly different than the one previously discussed. In order the expression to fit inside the table field, I’m using an abstract alternative.

(vSERVER.HEALTH = 50) || (S2_PRI || S2_BCK).STATE.NE(UP) - means (Virtual Server Health percentage equals 50 ) or (Site 1 PRIMARY or Site 2 BACKUP state not equal to UP)

The actual rule will looks like this:

"SYS.VSERVER(\"LB_VS_RGBAPP_S1_PRI_S2_BCK\").HEALTH.EQ(50)||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.99\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.100\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.106\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.105\",80).STATE.NE(UP)))"

What we discussed so far could be a working scenario even only with the Service Group expressions for STATE check - (S2_PRI||S2_BCK).STATE.NE(UP). The HEALTH expression was added, because of the requirement for minimum of two active Servers. This is it as individual servers could be failing inside the Service Group. We check the STATE of individual Server Group members, but we combine it to receive the Service Groups state as a whole, hence our expression will not detect individually failed services inside the Service Group. The HEALTH Expression will on other hand do exactly that.

Note: Percentage is based on Server’s configured weight. By default all servers have same weight.

The HEALTH expression on the first Virtual Server is slightly different than the rest. This is because when individual servers start failing we want to make sure our backup servers are available, before initiating next Virtual Server in chain. We have a fail safe logic for the Service Groups and we know we’ll always pick up best pairing combination for Service Group failure, but we lack the same for individual servers that’s why we need to be sure we can improve the situation by falling in chain. The “=” expression operator and the check for the Backup Server’s STATE protects us from situations where we have 1 Failed Server from Site 1 PRI & Site 1 BCK + 1 Failed Server from Site 2 PRI & Site 2 BCK. In such cases we won’t improve by falling in chain and it’s better to keep the Site 1 PRI & Site 2 PRI handling the traffic until Backup Server Group is fully available.

We neither have a fail safe logic for individual servers failing inside the Backup Server Groups after the Primary have already failed. This is the point where Tier 7 kicks in. For cases where we have 1 or 2 servers working from 1 or 2 Service Groups. We do not need to identify them, just place all 8 servers in Tier 7 Backup Load balancer. Once the other servers’ HEALTH or STATE return to normal the traffic will go back up the chain.

All NetScaler CLI commands for implementing or testing the setup are listed in the end of the post. The POC is with dummy IPs with disabled monitoring in the range of 169.254.10.0/24. I used Responder Policies containing Virtual Backup Server info so I know which one I’m hitting while testing the chain failover.

Remember there is no one way of doing things! That’s why in another series we’ll check how you can accomplish exactly the same with NetScaler Admin Partitions!

Nice To Have

I would like to point out a few improvements that could simplify such setups and gain flexibility.

Expression to check ServiceGroup Health & State without specifying the individual members. Same Virtual Server level expressions to be available on ServiceGroup level. This would greatly reduce the complexity of the expressions and increase the flexibility of the platform.

Spillover action that can steer traffic to Backup Virtual Load Balancer of our choice, not just the default one. Currently we have no control over the spillover Virtual Server direction. I would love to see a Spillover Action act much like a Content-Switching Action. Then we’ll have Rule Based Backup Virtual Servers.

# Config Snippet generated on xconfig.io
add server SRV_DUMMY_1 169.254.10.99
add server SRV_DUMMY_2 169.254.10.100
add server SRV_DUMMY_3 169.254.10.101
add server SRV_DUMMY_4 169.254.10.102
add server SRV_DUMMY_5 169.254.10.103
add server SRV_DUMMY_6 169.254.10.104
add server SRV_DUMMY_7 169.254.10.105
add server SRV_DUMMY_8 169.254.10.106
add serviceGroup LB_SG_RGBAPP_PRIMARY_S1 HTTP -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -healthMonitor NO -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
bind serviceGroup LB_SG_RGBAPP_PRIMARY_S1 SRV_DUMMY_2 80
bind serviceGroup LB_SG_RGBAPP_PRIMARY_S1 SRV_DUMMY_1 80
add serviceGroup LB_SG_RGBAPP_PRIMARY_S2 HTTP -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -healthMonitor NO -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
bind serviceGroup LB_SG_RGBAPP_PRIMARY_S2 SRV_DUMMY_4 80
bind serviceGroup LB_SG_RGBAPP_PRIMARY_S2 SRV_DUMMY_3 80
add serviceGroup LB_SG_RGBAPP_BACKUP_S1 HTTP -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -healthMonitor NO -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
bind serviceGroup LB_SG_RGBAPP_BACKUP_S1 SRV_DUMMY_6 80
bind serviceGroup LB_SG_RGBAPP_BACKUP_S1 SRV_DUMMY_5 80
add serviceGroup LB_SG_RGBAPP_BACKUP_S2 HTTP -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -healthMonitor NO -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
bind serviceGroup LB_SG_RGBAPP_BACKUP_S2 SRV_DUMMY_8 80
bind serviceGroup LB_SG_RGBAPP_BACKUP_S2 SRV_DUMMY_7 80
add lb vserver LB_VS_RGBAPP_ALL_SERVERS SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 -comment RGBAPP-Tier-7
bind lb vserver LB_VS_RGBAPP_ALL_SERVERS LB_SG_RGBAPP_PRIMARY_S1
bind lb vserver LB_VS_RGBAPP_ALL_SERVERS LB_SG_RGBAPP_PRIMARY_S2
bind lb vserver LB_VS_RGBAPP_ALL_SERVERS LB_SG_RGBAPP_BACKUP_S1
bind lb vserver LB_VS_RGBAPP_ALL_SERVERS LB_SG_RGBAPP_BACKUP_S2
set ssl vserver LB_VS_RGBAPP_ALL_SERVERS -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_ALL_SERVERS -certkeyName RGBAPP_KERNETIX_LOCAL
add lb vserver LB_VS_RGBAPP_S2_PRI_S2_BCK SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 -backupVServer LB_VS_RGBAPP_ALL_SERVERS -comment RGBAPP-Tier-6
bind lb vserver LB_VS_RGBAPP_S2_PRI_S2_BCK LB_SG_RGBAPP_PRIMARY_S2
bind lb vserver LB_VS_RGBAPP_S2_PRI_S2_BCK LB_SG_RGBAPP_BACKUP_S2
set ssl vserver LB_VS_RGBAPP_S2_PRI_S2_BCK -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_S2_PRI_S2_BCK -certkeyName RGBAPP_KERNETIX_LOCAL
add lb vserver LB_VS_RGBAPP_S1_PRI_S1_BCK SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 -backupVServer LB_VS_RGBAPP_S2_PRI_S2_BCK -comment RGBAPP-Tier-5
bind lb vserver LB_VS_RGBAPP_S1_PRI_S1_BCK LB_SG_RGBAPP_PRIMARY_S1
bind lb vserver LB_VS_RGBAPP_S1_PRI_S1_BCK LB_SG_RGBAPP_BACKUP_S1
set ssl vserver LB_VS_RGBAPP_S1_PRI_S1_BCK -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_S1_PRI_S1_BCK -certkeyName RGBAPP_KERNETIX_LOCAL
add lb vserver LB_VS_RGBAPP_S1_BCK_S2_BCK SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 -backupVServer LB_VS_RGBAPP_S1_PRI_S1_BCK -comment RGBAPP-Tier-4
bind lb vserver LB_VS_RGBAPP_S1_BCK_S2_BCK LB_SG_RGBAPP_BACKUP_S1
bind lb vserver LB_VS_RGBAPP_S1_BCK_S2_BCK LB_SG_RGBAPP_BACKUP_S2
set ssl vserver LB_VS_RGBAPP_S1_BCK_S2_BCK -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_S1_BCK_S2_BCK -certkeyName RGBAPP_KERNETIX_LOCAL
add lb vserver LB_VS_RGBAPP_S1_PRI_S2_BCK SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 -backupVServer LB_VS_RGBAPP_S1_BCK_S2_BCK -comment RGBAPP-Tier-3
bind lb vserver LB_VS_RGBAPP_S1_PRI_S2_BCK LB_SG_RGBAPP_PRIMARY_S1
bind lb vserver LB_VS_RGBAPP_S1_PRI_S2_BCK LB_SG_RGBAPP_BACKUP_S2
set ssl vserver LB_VS_RGBAPP_S1_PRI_S2_BCK -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_S1_PRI_S2_BCK -certkeyName RGBAPP_KERNETIX_LOCAL
add lb vserver LB_VS_RGBAPP_S1_BCK_S2_PRI SSL 0.0.0.0 0 -persistenceType NONE -cltTimeout 180 -backupVServer LB_VS_RGBAPP_S1_PRI_S2_BCK -comment RGBAPP-Tier-2
bind lb vserver LB_VS_RGBAPP_S1_BCK_S2_PRI LB_SG_RGBAPP_PRIMARY_S2
bind lb vserver LB_VS_RGBAPP_S1_BCK_S2_PRI LB_SG_RGBAPP_BACKUP_S1
set ssl vserver LB_VS_RGBAPP_S1_BCK_S2_PRI -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_S1_BCK_S2_PRI -certkeyName RGBAPP_KERNETIX_LOCAL
add lb vserver LB_VS_RGBAPP_S1_PRI_S2_PRI SSL 0.0.0.0 0 -persistenceType SOURCEIP -lbMethod LEASTRESPONSETIME -redirectURL "https://rgbapp.kernetix.local/maintenance" -cltTimeout 180 -backupVServer LB_VS_RGBAPP_S1_BCK_S2_PRI -comment RGBAPP-Tier-1
add cs action CS_ACT_RGBAPP -targetLBVserver LB_VS_RGBAPP_S1_PRI_S2_PRI
bind lb vserver LB_VS_RGBAPP_S1_PRI_S2_PRI LB_SG_RGBAPP_PRIMARY_S1
bind lb vserver LB_VS_RGBAPP_S1_PRI_S2_PRI LB_SG_RGBAPP_PRIMARY_S2
set ssl vserver LB_VS_RGBAPP_S1_PRI_S2_PRI -sslProfile MGDSVC_FE_MODERN_2020_Q4
bind ssl vserver LB_VS_RGBAPP_S1_PRI_S2_PRI -certkeyName RGBAPP_KERNETIX_LOCAL
add spillover policy SPILOVER_POL_TIER_1 -rule "(SYS.VSERVER(\"LB_VS_RGBAPP_S1_PRI_S2_PRI\").HEALTH.EQ(50)&&((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.104\",80).STATE.EQ(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.103\",80).STATE.EQ(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.106\",80).STATE.EQ(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.105\",80).STATE.EQ(UP))))||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.104\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.103\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.106\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.105\",80).STATE.NE(UP)))||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.99\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.100\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S2\").MEMBER(\"169.254.10.101\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S2\").MEMBER(\"169.254.10.102\",80).STATE.NE(UP)))" -action spillover
add spillover policy SPILOVER_POL_TIER_2 -rule "SYS.VSERVER(\"LB_VS_RGBAPP_S1_BCK_S2_PRI\").HEALTH.EQ(50)||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.104\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.103\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S2\").MEMBER(\"169.254.10.101\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S2\").MEMBER(\"169.254.10.102\",80).STATE.NE(UP)))" -action spillover
add spillover policy SPILOVER_POL_TIER_3 -rule "SYS.VSERVER(\"LB_VS_RGBAPP_S1_PRI_S2_BCK\").HEALTH.EQ(50)||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.99\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.100\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.106\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.105\",80).STATE.NE(UP)))" -action spillover
add spillover policy SPILOVER_POL_TIER_4 -rule "SYS.VSERVER(\"LB_VS_RGBAPP_S1_BCK_S2_BCK\").HEALTH.EQ(50)||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.104\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.103\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.106\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.105\",80).STATE.NE(UP)))" -action spillover
add spillover policy SPILOVER_POL_TIER_5 -rule "SYS.VSERVER(\"LB_VS_RGBAPP_S1_PRI_S1_BCK\").HEALTH.EQ(50)||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.99\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S1\").MEMBER(\"169.254.10.100\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.104\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S1\").MEMBER(\"169.254.10.103\",80).STATE.NE(UP)))" -action spillover
add spillover policy SPILOVER_POL_TIER_6 -rule "SYS.VSERVER(\"LB_VS_RGBAPP_S2_PRI_S2_BCK\").HEALTH.EQ(50)||((SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S2\").MEMBER(\"169.254.10.101\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_PRIMARY_S2\").MEMBER(\"169.254.10.102\",80).STATE.NE(UP))||(SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.106\",80).STATE.NE(UP)&&SYS.SERVICEGROUP(\"LB_SG_RGBAPP_BACKUP_S2\").MEMBER(\"169.254.10.105\",80).STATE.NE(UP)))" -action spillover
bind lb vserver LB_VS_RGBAPP_S1_PRI_S2_PRI -policyName SPILOVER_POL_TIER_1 -priority 100 -gotoPriorityExpression END -type REQUEST
bind lb vserver LB_VS_RGBAPP_S1_BCK_S2_PRI -policyName SPILOVER_POL_TIER_2 -priority 100 -gotoPriorityExpression END -type REQUEST
bind lb vserver LB_VS_RGBAPP_S1_PRI_S2_BCK -policyName SPILOVER_POL_TIER_3 -priority 100 -gotoPriorityExpression END -type REQUEST
bind lb vserver LB_VS_RGBAPP_S1_BCK_S2_BCK -policyName SPILOVER_POL_TIER_4 -priority 100 -gotoPriorityExpression END -type REQUEST
bind lb vserver LB_VS_RGBAPP_S1_PRI_S1_BCK -policyName SPILOVER_POL_TIER_5 -priority 100 -gotoPriorityExpression END -type REQUEST
bind lb vserver LB_VS_RGBAPP_S2_PRI_S2_BCK -policyName SPILOVER_POL_TIER_6 -priority 100 -gotoPriorityExpression END -type REQUEST