lec10
dz / distributed_systems_MIT / lec10Summary
Cloud replicated databases, Aurora
Node Tree
- RDS
- app_4_reps_complete
-
aurora
- amazon
- append_only_log_writes
- aurora_description
-
backhistory
-
ec2
- ebs
- good_for_webservers
- not_good_database
- physical_storage
-
ec2
- big_db_handling
- db_server
- db_tutorial
- quorum
- storage_server
- availability_zone
Nodes
aurora | |
content | Aurora |
children | amazon (Created by), append_only_log_writes, aurora_description (description), backhistory (Backhistory before Aurora), big_db_handling, db_server, db_tutorial (Fundamentals to understand Aurora architecture), quorum, storage_server |
amazon | |
content | Amazon |
parents | aurora |
aurora_description | |
content | Reliable, high performance, cloud databse |
parents | aurora |
backhistory | |
content | Backhistory |
children | ec2 |
parents | aurora |
ec2 | |
content | EC2 |
children | ebs (works with), good_for_webservers, not_good_database, physical_storage |
parents | backhistory |
good_for_webservers | |
content | Good for webservers |
children | not_good_database (not as good for) |
parents | ec2 |
not_good_database | |
content | Not as good for databses |
children | data_loss_on_hardware_crash (why ec2 wasn't good for datases) |
parents | good_for_webservers, ec2 |
physical_storage | |
content | Storage: physical, slice to guest |
children | data_loss_on_hardware_crash |
parents | ec2 |
data_loss_on_hardware_crash | |
content | If hardware crashes, data is lost |
children | ebs (fixes), s3 (intermediate solution) |
parents | not_good_database, physical_storage |
s3 | |
content | S3 |
parents | data_loss_on_hardware_crash |
ebs | |
content | EBS: elastic block store |
children | ebs_problems, looks_like_regular_drive, not_sharing_system, replicated_pair_storage_servers (description) |
parents | ec2, data_loss_on_hardware_crash |
replicated_pair_storage_servers | |
content | Replicated pair of storage servers |
parents | ebs |
not_sharing_system | |
content | Not a system for sharing |
parents | ebs |
looks_like_regular_drive | |
content | Looks like a regular drive to client |
parents | ebs |
ebs_problems | |
content | problems with EBS |
children | large_volumes_network_traffic, not_fault_tolerant |
parents | ebs |
not_fault_tolerant | |
content | Not very fault tolerant |
parents | ebs_problems |
large_volumes_network_traffic | |
content | Produces large volumes of network trafic |
children | database (I think this has to do with how databases send whole,pages around the network) |
parents | ebs_problems |
database | |
content | database |
children | db_tutorial |
parents | large_volumes_network_traffic |
availability_zone | |
content | Availability zone |
children | data_configuration, read_2_dead_az_1_other_server, write_one_dead_az |
db_tutorial | |
content | Database Tutorial |
children | crash_recovery, data_pages, locks, transaction, write_ahead_log |
parents | aurora, database |
remarks | A brief overview of how databases work. Just enough to understand Aurora |
transaction | |
content | Transaction |
children | locks, multi_ops_grouped_atomic |
parents | db_tutorial |
multi_ops_grouped_atomic | |
content | Multiple operations grouped and declared atomic |
children | locks, transactions_all_or_none |
parents | transaction |
locks | |
content | Locks |
parents | db_tutorial, multi_ops_grouped_atomic, transaction |
crash_recovery | |
content | Crash Recovery |
children | transactions_all_or_none |
parents | db_tutorial |
data_pages | |
content | Data pages |
parents | db_tutorial |
transactions_all_or_none | |
content | After crash: all transactions commited or none |
parents | crash_recovery, multi_ops_grouped_atomic |
write_ahead_log | |
content | Write-Ahead log (WAL) |
children | redo |
parents | db_tutorial |
redo | |
content | redo: do all transactions again using log |
parents | write_ahead_log |
RDS | |
content | RDS |
children | RDS_def |
RDS_def | |
content | Replicated database in multiple availability zones |
parents | RDS |
db_server | |
content | Database server |
children | custom_software, database_setup, storage_understand_db_entries |
parents | aurora |
custom_software | |
content | Custom software |
parents | db_server |
database_setup | |
content | Setup |
children | log_records |
parents | db_server |
data_configuration | |
content | Data: 6 replicants in 3 availability zones |
children | super_ft |
parents | availability_zone |
log_records | |
content | Log records anything being sent to replicas |
children | immediately_appended_log_records, log_smaller_than_pages |
parents | database_setup, storage_server |
log_smaller_than_pages | |
content | Log entries smaller than database pages |
parents | log_records |
storage_understand_db_entries | |
content | Underlying storage system understands database entries |
parents | db_server |
remarks | An important fact. Aurora went with a specialized filesystem rather than a generalized filesystem and got significant performance gains as a result of this. |
super_ft | |
content | Super fault-tolerant |
parents | data_configuration |
quorum | |
content | Quorum |
children | ft_goals, replication |
parents | aurora |
ft_goals | |
content | Fault-Tolerant Goals |
children | fast_replication, read_2_dead_az_1_other_server, ride_out_transient_slowness, storage_only, write_one_dead_az |
parents | quorum |
write_one_dead_az | |
content | Write with one dead availability zone (AZ) |
parents | availability_zone, ft_goals |
read_2_dead_az_1_other_server | |
content | Read with 2 dead availability zones plus one other server |
parents | availability_zone, ft_goals |
ride_out_transient_slowness | |
content | Ride out ransient slowness in replicas |
parents | ft_goals |
fast_replication | |
content | Fast replication |
children | parallel_copy |
parents | ft_goals |
storage_only | |
content | Storage Only |
parents | ft_goals |
replication | |
content | Replication |
children | N_rep, R_replicas, W_writes |
parents | quorum |
N_rep | |
content | N rep |
children | R_plus_W_equals_N_plus_one |
parents | replication |
W_writes | |
content | W Writes |
children | R_plus_W_equals_N_plus_one |
parents | replication |
R_replicas | |
content | R replicas |
children | R_plus_W_equals_N_plus_one |
parents | replication |
R_plus_W_equals_N_plus_one | |
content | R + W = N + 1 |
children | guaranteed_overlap, variable_defaults |
parents | N_rep, R_replicas, W_writes |
guaranteed_overlap | |
content | Guaranteed to overlap |
parents | R_plus_W_equals_N_plus_one |
variable_defaults | |
content | N=6,W=4,R=3 |
parents | R_plus_W_equals_N_plus_one |
remarks | Defaults used by the Aurora setup |
append_only_log_writes | |
content | Append-only log writes |
children | Appendtoatleast4replicastobecompleted |
parents | aurora |
app_4_reps_complete |
Appendtoatleast4replicastobecompleted | |
parents | append_only_log_writes |
parallel_copy | |
content | Parallel Copy |
children | chunks_of_log_protection_groups |
parents | fast_replication |
chunks_of_log_protection_groups | |
content | Chunks of log protection groups |
parents | parallel_copy |
storage_server | |
content | Storage Server |
children | db_tracks_server_log_indexes, log_records |
parents | aurora |
apply_log_record_to_page | |
content | Apply log record to page when ready req |
parents | old_page_log_records |
remarks | not sure what I meant by "req" also, added that this is "quorum logic" not here |
old_page_log_records | |
content | Old page and log records |
children | apply_log_record_to_page |
parents | immediately_appended_log_records |
immediately_appended_log_records | |
content | Immediately appened to list of log records |
children | old_page_log_records |
parents | log_records |
db_tracks_server_log_indexes | |
content | DB tracks server log indexes (how far along) for each server |
children | db_read_updated_ss |
parents | storage_server |
db_read_updated_ss | |
content | On read, find up-to-date storage server and read from that |
children | when_quorum_reads_used |
parents | db_tracks_server_log_indexes |
when_quorum_reads_used | |
content | Quorum reads used during crash recovery of DB server |
parents | db_read_updated_ss |
big_db_handling | |
content | Big Database Handling |
children | sharding |
parents | aurora |
sharding | |
content | sharding |
children | protection_servers |
parents | big_db_handling |
protection_servers | |
content | Protection Servers |
children | groups_of_six_servers, how_to_split_up_logs, logs, pg_stores_logs_relevant_to_pages |
parents | sharding |
logs | |
content | Logs |
parents | protection_servers |
how_to_split_up_logs | |
content | How to split up logs? |
children | pg_stores_logs_relevant_to_pages |
parents | protection_servers |
pg_stores_logs_relevant_to_pages | |
content | Protection Group stores logs relevant to pages |
children | find_pg_with_data |
parents | how_to_split_up_logs, protection_servers |
groups_of_six_servers | |
content | Groups of six servers |
parents | protection_servers |
find_pg_with_data | |
content | Find protection group with data |
parents | pg_stores_logs_relevant_to_pages |