Table of Contents

Description

This will NOT be a 5 min reading article.
It will require more time but it will cover all areas of interest :)
Most importantly, at the end, your Setup will WORK.

Kudos to Adam, my colleague, who first tested this in his lab.
It served as a basis for both me and the customer later on.

To start things off, a little while back someone asked me how he can have:
Incoming Traffic from Outside -> Azure Aviatrix Environment with FW Inspection -> Spoke with his Application
and also Preserve the source IP address when it hits the FW Rules.

Why this ?

He wanted to use the real source IP for logging and filtering purposes on Aviatrix FireNet attached FWs.
Pretty understandable and reasonable.

He read our article from here:
Spoke Ingress with Application Gateway
He saw the Diagram:

but noticed that the Azure Application Gateway does SNAT and as such his Firenet Firewalls are not seeing the original User IP of the request.

One could argue that the X-Forwarded-For header added by the Azure Application Gateway can be used to preserve the original IP…

Do you see a corner case here ?
I for one did not at the beginning. I always rush through things.

What if your Application is NOT a WebApp and as such there is NO HTTP Header where to add the X-Forwarded-For to?

A High-Level overview of what I was trying to achieve for him:

User/Internet –> Load Balancer –> Firewalls –> Aviatrix Transit –> Aviatrix Spokes –> Application facing internal LB –> Application VMs

The Ingress Spoke scenario did not match his needs because of the SNAT on the Application facing Load Balancer inside the Spoke.
You can read more about that setup here:
Spoke Ingress with Application Gateway
The flow in this case is:

Client → Spoke + Ingress LB → Transit → Spoke Destination → VM(s) of Service to be reached The Ingress/External LB does SNAT => the Source IP from the Internet User is lost.

Why is this done by the AppGW?

Without it there would be no way to return the traffic to the Ingress spoke, especially if one has an Egress Transit (spoiler: Egress Transit originates 0/0 toward Spokes).

The need to see the source IP of ingress traffic opens the door to 2 (or rather 3) distinct possibilities:

  • Scenario 1:
    This is the focus of this article. Easy to expand to Scenario 2.

    Service is hosted on a single VM, behind a Spoke, Firewall receives traffic destined for external LB VIP (one VIP <–> one Service <–> 1 x VM behind Spoke)


    Traffic FLOW:

    • Ingress in Firenet VNET → external Load Balancer with VIP/service → reroute VIP destined traffic with NO-SNAT to the Firewalls (as backends).
    • Firewalls receive traffic destined to VIP → DNAT to VM behind Spoke hosting the service.
    • Firewalls SNAT to Firewall IP on sending the packet out (to ensure the traffic returns on the same Firewall and is not dropped because of a non-existing session if it were to land on the other firewalls.
    • Firewalls have NO session sync in the Cloud unfortunately.

    This ReRoute VIP to the Firewall is what Azure calls and offers as Checkbox in the Ingress Rule LB definition: “Use Floating IP”


  • Scenario 2:

    Service is hosted on a multiple VMs, behind a Spoke.
    There is an internal LB behind that Spoke that points to those VMs.
    Firewall receives traffic destined for external LB VIP (one VIP <–> one Service <–> 1 x internal LB behind Spoke) <–> 1..n x VMs behind the internal LB).
    In this case the Firewall does:

    • DNAT to Internal Load Balancer (behind Spoke) FrontEnd IP
    • SNAT to FW IP so that the return traffic comes back to the same Firewall

  • Scenario 3 (NOT covered here but similar)

    Ingress LB receives traffic destined for VIP, each service listens on a different port.
    Web Service for Something. is on VIP: 80.
    Web Service for Something Else Strategies is on VIP : 81.
    And the list of ports goes on: 82, 83…65535.


    Traffic FLOW:

    • Firewall receives traffic destined for a certain port to it as backend (destination is no longer the VIP).

    • Depending on the port it DNATs it either to a single VM (if the service is only on a single VM)

      OR

    • to the FrontEnd IP of an internal load balancer behind a Spoke in case the service is hosted on multiple VMs.

    • It does SNAT also on sending the traffic out to ensure session persistence to the same firewall (asymmetry would cause traffic drops because of no session sync).

      NO FLOATING IP on the Ingress LB.

Deployment

Logical Steps

  • create an external facing LB (or NLB depends how you want to name it..this is not an article about phonetics:) )
  • define a VIP
  • a backend = the Firewalls
  • healthcheck = TCP on port 443

    NO FLOATING IP on the Ingress LB.

  • define ONLY an INGRESS rule (no Outbound rule) in which you enable this Floating IP checkbox (will not do any NAT as a consequence)
  • on the Firewalls
    • enable HTTPs access on eth1/1 (WAN / egress Interface)
    • enable on the virtual router having all your interfaces:
      • ECMP
      • symmetric return (Azure LB both the Firenet one and the Ingress one send HealthCheck from 168.63.129.16/32 which arrive on both FW interfaces => we need to reply where they arrive)
  • ensure HealthCheck packets from 168.63.129.16/32 incoming on eth1/1 & eth1/2 (Internet / AVX TGW facing) are allowed by Policy
    In my Case: Management Profile + specific Firewall rule if still not working
  • add SNAT rule for traffic exiting eth1/2 (toward our gateways) and not being RFC1918 (no internally originated Spoke traffic from the Firenet case E-W inspection should get SNATed)
  • add DNAT rule for traffic having entered eth1/1 (external) to go to either your VM behind a spoke
    OR if the service is hosted on Multiple VMs then to the App Facing Load Balancer FrontEnd IP behind the Spoke that has those VMs as Backends (Scenario 2)

If asking yourself why ECMP/Symmetric Return and the /32 route…then see below:

  • our integration adds RFC 1918 routes toward eth1/2
    (facing the AVX Transit Gateway - FW interconnect network)
  • it also adds a route for 168.63.129.16 (Microsoft LoadBalancer healthcheck SOURCE IP used to see if the Firewalls are alive)
  • this 168.63.129.16/32 route is added by our integration on eth1/2
  • when adding this extra External Ingress Load Balancer..then the Firewalls will start receiving keepalives from 168.63.129.16 on eth1/1 and thus reply as per the static route on eth1/2 (never reaching back the load balancer and being a massive FAIL)
  • solution is to:

    • add 168.63.129.16/32 via nexthop (azure router IP of the subnet) on eth1/1

    • ECMP checkbox under Virtual Router makes this possible.

      Enabling symmetric return makes it that:

      • when a healthcheck comes from the Firenet E-W load balancer (on eth1/2) then the return packet for it is sent on eth1/2
        &
      • when the healtcheck comes from this NEW INGRESS LB on eth1/2 then the reply for it it send out via eth1/2

Implementation

This one covers just the Ingress/Egress LB part.
The Spoke LB if you have a service hosted on multiple VMs is not covered and is also pretty basic to do.
You just have to change the DNAT rule on the Firewall at the end of the article to point to the FrontEnd IP of your Spoke LB instead of directly the service VM IP.

Create Load Balancer

Add BackEnd Pool

Take Firewall IPs facing Ingress/Egress either:

  • from the Palo Mgmt interface → eth1/1 → click on it and see what IP it got via DHCP
  • from Azure Portal → Resource Groups → the Ingress/Egress one from Transit Firenet

Make sure you select the Virtual Network: Egress one of our Transit Firenet:

Define the Floating IP Inbound rule (NO OUTBOUND rule is needed)

This will forward VIP destined traffic to the firewalls without doing any NAT and with maintaining Dest_IP = VIP (it’s just some sort of quirky PBR).

LOAD Balancing Rule, NOT Inbound NAT rule

Define HealthCheck on port 443 as TCP and NOT HTTPs

Get Info from Azure about the Load Balancer

Click on Insights and look at this schema.
The lines from the backend pool to the FWs (in this case nazufrcepanavx0[1-2]) should be GREEN.

Alternatively go to Metrics → choose Health Probe

The graph that appears should not be a flat line to 0.

Firewall - make sure you do not get 2 x 0/0 from DHCP

Firewall - add ECMP and symmetric return

Firewall - add LB healthcheck IP /32 via eth1/1 (Egress Interface)

Management Profile (permit HTTPs on eth1/1)



For extra safety you can add there under Permitted IPs for HTTPs the LB healtcheck IP (168.63.129.16).
Microsoft always uses this one for LBs sending out Health Checks Add also your Panorama IP + your own computer if you intend to connect directly to the Firewall to manage it.

or just define your own service, tcp port 443
(might be cleaner than using the predefined ones)


Policy Rule to allow Inbound TCP 443

SNAT/DNAT Rules

MAKE SURE YOU HAVE A ROUTE FOR 0/0.

IF THE FW GETS PACKETS WITH DST = PUBLIC VIP AND IT HAS NO ROUTE TO DO RPF ON IT => IT DROPS THE PACKET BEFORE HITTING THE POLICY.
# On Palo do this periodically while generating traffic

show counter global filter severity drop

name                                   value     rate severity  category  aspect    description
--------------------------------------------------------------------------------
flow_policy_nofwd                        247        0 drop      flow      session   Session setup: no destination zone from forwarding

If you see this => route missing :slight_smile:

DEBUG on FW

  • Packet Capture


  • Session Filters 


    localadmin@nazufrcepanavx02> show session all filter destination <VIP Ingress LB>


--------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
7345 undecided ACTIVE FLOW <User accessing App IP>[41431]/WAN/6 (<User accessing App IP>[41431])
vsys1 <VIP Ingress LB>[22]/WAN (<VIP Ingress LB>[22])

    localadmin@nazufrcepanavx02> show session all filter destination <VIP Ingress LB>



--------------------------------------------------------------------------------
ID Application State Type Flag Src[Sport]/Zone/Proto (translated IP[Port])
Vsys Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
7464 ssh ACTIVE FLOW NB <User accessing App IP>[65067]/WAN/6 (10.249.30.132[46261])
vsys1 <VIP Ingress LB>[22]/LAN (10.208.4.5[22])