It is a physical fibre connection through which we can access AWS services without sending traffic through the public internet
The connection is between the business premises => DX Location => AWS Region
When we order a DX connection, what we order is actually a Port Allocation at a DX Location
The port has 2 costs:
Hourly cost based on the DX location and speed of the port
Outbound data transfer, inbound data transfer is free of charge
In order to get a Direct Connect we have to create a connection (logical entity) inside an AWS account. This connection will have an unique ID
Provisioning time: weeks/months of extra time
DX provides low and consistent latency + high speeds (1/10/100 Gbps)
In the DX location we will have to install a cross-connect (physical cable) in order to connect our own network to the AWS network
DX Physical Connection Architecture
A Direct Connect is a physical port allocated at a DX Location
This physical port provides 1, 10 or 100 Gbps speed
DX can be ordered directly from AWS or through partners (wider range of speeds, with less options)
The port requires single-mod fibre, NO copper cable connection supported
Depending on the speed we order we will interface with DX using the following standards:
1000BASE-LX (1310 nm) Transreceiver for 1 Gbps
10GBASE-LR (1310 nm) Transreceiver for 10 Gbps
100GBASE-LR4 - 100 Gbps
Auto-Negotiation should be disabled for the connection
We configure the port speed, full-duplex should be manually set on the network connection
The router in the DX location should support Border Gateway Protocol (BGP) and BGP MD5 Authentication
Optional configurations:
MACsec
Bidirectional Forwarding Detection (BFD)
Direct Connect - MACsec
It is a security feature that improves/partially improves a long-standing problem with DX: lack of builtin encryption
It is a standard which allows frames on the network to be encrypted. Frames are the unit of data which occur at the layer 2 of the OSI model
MACsec provides a hop by hop encryption architecture => 2 devices need to be next to each other at layer 2 to order MACsec (layer 2 adjacency)
MACsec features:
Confidentiality: strong encryption at layer 2 by encryption the frame’s EtherType and payload
Data integrity: adds additional fields to ensure that data cannot be modified in transit without both parties being able to detect the modification
Data origin authenticity: both parties can see that frames were been sent by other trusted peer
Replay protection
MACsec does not replaces IPSEC over DX, it is not end-2-end!
It is designed to allow transfer for super high speeds for terabit networks
MACsec key components:
Secure Channel (unidirectional): each MACsec participant creates a MACsec channel that is used to send traffic
Secure Channels are assigned an identifier (SCI): uniquely identifies a secure channel
Secure Associations: communication that occurs on each secure channel, takes place as a series of transient sessions, multiple secure associations will take place on each secure channel over the lifetime of the connection. Each secure channel generally has 1 secure association at a time (exception when the associations are being replaced)
MACsec encapsulation: 16 bytes MACsec tag & 16 bytes of Integrity Check Value (ICV). MACSec modifies Ethernet frames by inserting these tags
MACSec Key Agreement protocol: manages discovery, authentication and key generation
Cipher Suite: controls how the data is encrypted: algorithm, packets per key, key rotation
MACsec can be defined either ona DX connection on a Link Aggregation Group (LAG)
Configuring MACsec: we associate a CAK/CKN pair with the connection on both the AWS DX router(s) and customer side’s router
It is possible to extend MACsec from the DX location ot the customer side
This requires a dedicated physical extension of the cross connect to the business premises; this type of extension requires Layer 2 Adjacency
DX Connection Process
A DX connection begins in a DX Location, which contains AWS equipment and also customer/provider equipment
AWS does not own this facility (neither does the provider) - it is a data center owned by a third party
A large data center is collection of cages, these cages are areas that specific customers rent
Only the staff at the data center can connect stuff together, only they have access to the space in-between the cages to connect different cages together
Connecting these cages can be done only by staff members only when they have authorization from all parties
The authorization is called Letter of Authorization Customer Facility Access (LOA-CFA)
It is form that gives the access from one customer to get the data center staff to connect to the equipment of another customer
DX connection process:
DX Virtual Interfaces BGP Sessions + VLAN
DX Connections are a layer 1 connections (physical cables) running layer 2 protocols (Data Link)
We need a way to connect to multiple types of layer 3 (IP) networks (VPCs and public zones) over a single DX connection => this is where Virtual Interfaces (VIFs) come in handy
VIFs allows us to run multiple layer 3 networks over a layer 2 direct connect
A VIF is simply a BGP Peering Session => something which exchanges prefixes which allow traffic to be router to one point to the other
All of this is isolated within a VLAN
A VLAN isolates different layer 3 networks using VLAN tagging
BGP exchanges prefixes, which means each end knows which networks are at each side; BPG MD5 authentication means that only authenticated data will accepted by either type
3 types of VIFs can be run over DX:
Private VIF: used to connect ot AWS private networks, so VPCs
Public VIF: used to connect to Public Zone Service
Transit VIF: allow integration between DX and Transit Gateways
A single dedicated DX can have in total 50 public/private VIFs, as well as 1 Transit VF
Hosted VIFs: we can share VIFs with other AWS accounts. Can be connected to a virtual private gateway in a VPC of the other account
Private VIFs
They are used to access the resources inside 1 AWS VPC using private IP addresses
Resources can be accessed with their private IP using private VIFs, public IPs and Elastic IPs wont work
Private VIFs are associated with a Virtual Private Gateway (VGW) which can be associated to 1 VPC. This has to be in the same region where the DX location connection terminates
1 Private VIF = 1 VGW = 1 VPC (there are ways around this using Transit VIFs)
There is no encryption on private VIFs, DX is not adding encryption and neither is the private VIFs (there are ways around this, example using HTTPS)
With private VIFs we can use normal or Jumbo Frames (MTU of 1500 or 9001)
Using VGW, route propagation is enabled by default
Creating private VIFs:
Pick the connection the VIF will run over
Chose VGW (default) or Direct Connect Gateway
Chose who owns the interface (this account or another account)
Choose a VLAN id - 802.1Q which needs to match the customer config
We need to enter the BGP ASN of on-premises (public or private). If private use 64512 to 65535
We can choose IPs or auto generate them
AWS will advertise the VPC CIDR range and the BGP Peer IPs (/30)
We can advertise default or specific corporate prefixes (max 100 - this is HARD limit, the interface will go into an idle state)
Private VIFs architecture:
Key learning objectives:
Private VIFs are used to access private AWS services
Private VIF => 1 VGW => 1 VPC
VPC needs to be in the same region as the DX location
VGW has an AWS assigned
Over the private VIF runs the BGP with IPv4 or IPv6 (separate BPG peering connections)
We configure our own AS on the VIF, which can be private ASN or public ASN
Public VIFs
Are used to access AWS public zone services: both public services and services which have a public Elastic IP
They offer no direct access to private VPC services
We can access all public zone regions with one public VIF across AWS global network
AWS advertises all AWS public IP ranges to us, all traffic to AWS services will go over the AWS global network
We can advertise any public IPs we own over BGP, in case we don’t have public IPs, we can work with AWS support to allocate some to us
Public VIFs support bi-directional BGP communities
Advertised prefixes are not transitive, our prefixes don’t leave AWS
Create a public VIF:
We pick the connection the VIF will run over
We chose the interface owner (this account or another)
Chose VLAN - 802.1Q, which needs to match the customer configuration
Chose the customer side BGP ASN (ideally this is public ANS for full functionality offered by public VIFs)
Configure MD5 authentication and select optional peering IP addresses
We have to select which prefixes we want to advertise
Public VIFs architecture:
Direct Connect Public VIF + VPN
Using a VPN gives us an encrypted and authenticated tunnel
Architecturally, having a VPN over DX uses a Public VIF + VGW/TGW public endpoints
With a VPN we connect to public IPs which belong to a VGW or TGW
A VPN is transit agnostic: we can connect using a VPN to VGW or a TGW over the internet or over DX
A VPN is end-to-end encryption between a Customer Gateway (CGW) and TGW/VGW, while MACsec is single hop based
VPNs have wide vendor support
VPNs have more cryptographic overhead compared to MACsec
A VPN can be provided immediately, can be used while DX is provisioned and/or as a DX backup
Direct Connect Gateways
Direct Connect is a regional service
Once a DX connection is up, we can use public VIFs to access all AWS Public Services in all AWS regions
Private VIFs can only access VPCs in the same AWS regions via VGWs
Direct Connect Gateway is a global network device: it is accessible in all regions
We integrate with it on the on-premises side by creating a private VIF and associate this with a DX Gateway instead of the Virtual Private Gateway (VGW). This integrates the on-premises router with the DX Gateway
On the AWS side we create VGW associations in any VPC in any AWS regions
DX gateways allow to route through them to the on-premises environments and vice-versa. They don’t allow VPCs connected to the gateway to communicate with each other
We can have 10 VGW attachments per DX Gateway
1 DX connection can have up to 50 private VIFs, each of which support 1 DX gateway and 1 DX gateway supports 10 VGW association => we can connect up to 500 VPCs
DX gateway don’t have a cost, we have cost for data transit only
Cross-account DX Gateways: multiple account can create association proposal for a DX gateway
Transit VIFs and TGW
A DX Gateway does not route between the associated VPCs to that gateway, it only routes from on-premises to AWS side or vice-versa
Transit Gateways are regional, it is possible to peer TGWs allowing connections between regions
Transit Gateways are hub-and-spoke architecture, anything associated with a TGW is able to communicate with anything other associated to that TGW
This architecture also works within peered TGWs
DX-TGW Architecture:
A DX supports up to 50 public and private VIFs and only 1 Transit VIF
We can connect up to 3 Transit Gateways to a Direct Connect Gateway
An individual DX gateway can be used with VPCs and private VIFs or with Transit Gateways and transit VIFs, NOT BOTH at the same time!
Consideration:
DX gateway does not route between its attachments, this is why the peering connection between TGWs is required
Each TGW can be attached up to 20 DX gateways
Each TGW supports up 5000 attachments, up to 50 peering attachments
DX Gateway routing problems:
DX gateway only allows communications from a private VIF to any associated virtual private gateways
With a transit gateway we can solve this, if we connect the DX gateway to a transit gateway (works only in one region)
Direct Connect Resilience and HA
To improve resilience:
Order 2 DX ports instead of one => 2 cross connects, 2 customer DX routes connecting to 2 on-premises routes
Connect to 2 DX locations, have to customer routers and 2 on-premises routers in different buildings (geographically separated)
Not resilient DX architecture:
Resilient DX architecture:
Improved resilient DX architecture:
Extreme resilient DX architecture:
Direct Connect Link Aggregation Groups (LAG)
LAG: allows to take multiple physical connections and configure them to act as one
From speed perspective a LAG can have:
2 ports, each 100 Gbps
4 ports, the speed of each being less than 100 Gbps
We can create a LAG with the maximum speed of 200 Gbps
LAG do provide resilience, although AWS does not market them as such. They do not provide any resilience regarding hardware failure or the failure of entire location
LAGs use an Active/Active architecture, maximum 4 connection can be part of the LAG
All connections must have the same speed and terminate at the same DX location
A LAG has an attribute called minimumLinks: the LAG is active as long as the number of working connections is greater or equal to this value
