Hello,

Thank you for contacting AWS Premium Support. My name is Sumir and I will be assisting you with this case.

I understand your use case here and can understand that the Spoke to Spoke connection fails over your Hub VPC's (HUB VPC - NOC and test VPC).

Also, I believe your setup is something like a VPN Cloudhub architecture as described in the below link:

http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPN_CloudHub.html

This architecture should also work and I wanted to confirm if your different spoke offices are using non-overlapping networks? and to troubleshoot your current issue then please share the below details:

a.) Routing table and BGP table of the Office networks and the VPC details. Office to Office subnet details.
b.) Traceroute output from both offices for office to office communication.
c.) Packet captures on the source and destination when sending traffic between office to office.

AWS also recommends a similar architecture using a Transit VPC architecture as detailed in the below AWS blog and solutions document:

https://aws.amazon.com/blogs/aws/aws-solution-transit-vpc/

https://docs.aws.amazon.com/solutions/latest/cisco-based-transit-vpc/welcome.html

This architecture uses Cisco CSR1000v router to setup dynamic tunnels between the remote spoke offices with the hub VPC and is the recommended design. Let me know if you have any questions after going through the guide.

With respect to your other queries, I have mentioned the answers inline to the questions:

What is the best practice and recommendation from AWS for connecting multiple office to multiple VPCs?

If we still want to use BGP is it possible to stop routing from office to office via AWS? If we change the routing to static from each office and each VPC will this stop connection from office to office?

Answer:- If you want to avoid Office to Office traffic from going over AWS network then you can either use Transit VPC (Use DMVPN) or have a full mesh VPN between your Office to Office networks.

We use different private ASN for each office to connect to AWS. What is the difference between using same private ASN for all office and different ASN for each office?

Answer:- If you use same ASN between different sites when using BGP, then a route for a subnet/network advertised from one Office will not be accepted in another offices BGP table by default. This is because of BGP's AS path attribute to prevent loops (Section 5.1.2. - https://www.ietf.org/rfc/rfc4271.txt)

Please do not hesitate to let me know in case of any queries or issues.

Looking forward to your response.

Untitled