Infra Request for aprned & aprnes
h1. Request Details
h2. Background
We are going to decommission NE as ne-app has become single point of failure especially for hotel product. We had some incidents due to ne-app inability to handle traffic. Ne-app is serving too many things and most of them are accom data related. Accommodation product services depend to ne-app to get geo data, hotel review, and some accom data. We will move all accom services from NE to a separate service. Local team will also move their services out from ne-app.
We will move accom services from ne-app to aprned-app and aprnes-app.
- aprned-app will be the “provider data service” for accom data related to named entity, with the responsibility to handle all retrieve and update requests for these data without further modification (raw data CRUD requests) and job/schedule to update accom data. This service will become the only owner of accom data with direct access to accom databases (hotel-mongod, acontent-mongod, hotelasset-mongod, hroom-mongod) and related data providers.
- aprnes-app will serve application layer access of accom data (with further search/modification/computation as required), with dependency to aprned-app for supplying the raw data. As all the application layer mechanisms (e.g. caching, search filters, generated data computations) are implemented on this service, all services will connect to aprnes-app instead of aprned-app to get accom data.
Accom geo service will be in aprnes-app. The common geo service will be owned by local team. Thus aprnes-app will need to connect to locgs-app to get common geo data.
Traffic estimation for most accessed data:
- Geo search : 2500/s
- Hotel SEO : 2500/s
- Hotel review : 2025/s
- Hotel data : 400/s
h2. Purpose
- To reduce single point of failure service
- To have clear separation between accom and local services. This will reduce impact to other products in case there's any problem in hotel services.
h2. Impact
- Offload accom geo search service to separate cluster
- Reduce dependency between hotel and other products that use geo and landmark search
- We've hold some projects that need ne-app due to ne freeze. By separating it into separate clusters, we can continue those feature projects. The urgent feature is hotel review with tag feature.
h2. Risk
- Increase traffic to locgs-app by max 2000/s but aprnes-app will add cache for geo data
- Increase load on shared databases: data-mongod but it will be low as it's needed to get currency info data for generating hotel policy
h2. Resources
h3. EC2
h4. Configuration
{code}
count = "8"
instance_type = "c4.xlarge"
ebs_optimized = "false"
disable_api_termination = "false"
root_block_device = {
volume_type = "gp2"
volume_size = "8"
delete_on_termination = "true"
}
tags = {
Service = "aprned"
Cluster = "aprned-app"
ProductDomain = "apr"
Team = "apr"
Application = "java-7"
Environment = "production"
Description = "Accommodation Named Entity Data Service"
}
{code}
h4. EBS Volumes
-
h4. Additional Requirements
-
h3. EC2
h4. Configuration
{code}
count = "8"
instance_type = "c4.2xlarge"
ebs_optimized = "false"
disable_api_termination = "false"
root_block_device = {
volume_type = "gp2"
volume_size = "8"
delete_on_termination = "true"
}
tags = {
Service = "aprnes"
Cluster = "aprnes-app"
ProductDomain = "apr"
Team = "apr"
Application = "java-7"
Environment = "production"
Description = "Accommodation Named Entity Search Service"
}
{code}
h4. EBS Volumes
-
h4. Additional Requirements
-
h3. CLB
h4. Configuration
{code}
name = "aprned-lbint-01"
security_groups = "aprned-lbint"
internal = "true"
cross_zone_load_balancing = "true"
idle_timeout = "60"
connection_draining = "true"
connection_draining_timeout = "300"
listener = {
instance_port = "61022"
instance_protocol = "HTTP"
lb_port = "443"
lb_protocol = "HTTPS"
}
health_check = {
healthy_threshold = "10"
unhealthy_threshold = "2"
target = "HTTP:61022/healthcheck"
interval = "30"
timeout = "5"
}
tags = {
Name = "aprned-lbint-01"
Service = "aprned"
ProductDomain = "apr"
Team = "apr"
Environment = "production"
Description = "Internal Load Balancer for Accom Named Entity Data Service"
}
{code}
h4. CLB Attachment
aprned-app
h4. CNAME record
aprned.main.tvlk.cloud
h4. Additional Requirements
-
h3. CLB
h4. Configuration
{code}
name = "aprnes-lbint-01"
security_groups = "aprnes-lbint"
internal = "true"
cross_zone_load_balancing = "true"
idle_timeout = "60"
connection_draining = "true"
connection_draining_timeout = "300"
listener = {
instance_port = "61024"
instance_protocol = "HTTP"
lb_port = "443"
lb_protocol = "HTTPS"
}
health_check = {
healthy_threshold = "10"
unhealthy_threshold = "2"
target = "HTTP:61024/healthcheck"
interval = "30"
timeout = "5"
}
tags = {
Name = "aprnes-lbint-01"
Service = "aprnes"
ProductDomain = "apr"
Team = "apr"
Environment = "production"
Description = "Internal Load Balancer for Accom Named Entity Search Service"
}
{code}
h4. CLB Attachment
aprnes-app
h4. CNAME record
aprnes.main.tvlk.cloud
h4. Additional Requirements
-
h3. ElastiCache Memcached
h4. Sizing
- Avg number of objects : 250.000
- Peak number of objects : 250.000
- Avg object size : 1000 bytes
- Max object size : 1500 bytes
- Avg read count per sec : 2500
- Peak read count per sec : 5000
h4. Configuration
{code}
cluster_id = "aprgeo-memcached"
engine = "memcached"
engine_version = "1.4.x"
maintenance_window = "Mon
:18:00-Mon
:19:00"
node_type = "cache.m4.large"
num_cache_nodes = "2"
port = "11211"
parameter_group_name = "default.memcached1.4"
apply_immediately = "false"
az_mode = "cross-az"
{code}
tags = {
Name = "aprgeo-memcached"
Service = "aprnes"
ProductDomain = "apr"
Team = "apr"
Environment = "production"
Description = "Cache for accom geo summary"
}
{code}
h4. Custom Parameters
-
h3. ElastiCache Memcached
h4. Sizing
- Avg number of objects : 400.000
- Peak number of objects : 3.500.000
- Avg object size : 750 bytes
- Max object size : 2000 bytes
- Avg read count per sec : 2025
- Peak read count per sec : 4000
h4. Configuration
{code}
cluster_id = "aprrd-memcached"
engine = "memcached"
engine_version = "1.4.x"
maintenance_window = "Mon:18:00-Mon:19:00"
node_type = "cache.m4.large"
num_cache_nodes = "2"
port = "11211"
parameter_group_name = "default.memcached1.4"
apply_immediately = "false"
az_mode = "cross-az"
{code}
tags = {
Name = "aprrd-memcached"
Service = "aprnes"
ProductDomain = "apr"
Team = "apr"
Environment = "production"
Description = "Cache for accom review data"
}
{code}
h4. Custom Parameters
-
h3. Connectivity
h4. Rules
[^connectivity.csv]