Transcript
Exadata: from Beginner to Advanced in 3 Hours Arup Nanda Longtime Oracle DBA (and now DMA)
Why this Session? • If you are – an Oracle DBA • Familiar with RAC, 11gR2 and ASM
– about to be a Database Machine Administrator (DMA)
• How much do you have to learn? • • • •
How much of you own prior knowledge I can apply? What’s different in Exadata? What makes it special, fast, efficient? Do you have to go through a lot of training? Exadata: from Beginner to Advanced in 3 Hours
2
What is Exadata • It is like an appliance containing – Storage, Flash Disks, Database Servers, Infiniband Switches, Ethernet Switches, KVM (some models)
• But it is not an appliance. Why? – additional software to make it a better database machine – Components can be managed independently
• That’s why Oracle calls it a Database Machine (DBM) • And DMA – Database Machine Administrator Exadata: from Beginner to Advanced in 3 Hours
3
Anatomy of an Oracle Database Instance Combination of •Memory Areas •Background Processes
SELECT NAME FROM CUSTOMERS WHERE STATUS ='ANGRY' UPDATE CUSTOMERS SET BONUS = 1M WHERE STATUS ='ANGRY'
datafile1 datafile2
Storage
Exadata: from Beginner to Advanced in 3 Hours
4
RAC Database
datafile1 datafile2
Storage
Exadata: from Beginner to Advanced in 3 Hours
5
Query Processing
JILL SELECT NAME FROM CUSTOMERS WHERE STATUS ='ANGRY'
datafile1 datafile2
Database Block
Storage
Exadata: from Beginner to Advanced in 3 Hours
6
Components for Performance CPU Memory Network
Less I/O = better performance
I/O Controller Disk Exadata: from Beginner to Advanced in 3 Hours
7
What about SAN Caches? • Success of SAN caches is built upon predictive analytics • They work well, if a small percentage of disk is accessed most often – The emphasis is on disk; not data
• Most database systems – are way bigger than caches – need to get the data to the memory to process --> I/O at the disk level is still high
• Caches are excellent for filesystems . or very small databases Exadata: from Beginner to Advanced in 3 Hours
8
What about In-Memory DBs • Memory is still more expensive • How much memory is enough? • You have a 100 MB database and 100 MB buffer cache • The whole database will fit in the memory, right? • NO! • Oracle database fills up to 7x DB size buffer cache http://arup.blogspot.com/2011/04/can-i-fit-80mb-database-completely-in.html
Exadata: from Beginner to Advanced in 3 Hours
9
The Solution • A typical query may: – Select 10% of the entire storage – Use only 1% of the data it gets
• To gain performance, the DB needs to shed weight • It has to get less from the storage . Filtering at the storage level . The storage must be cognizant of the data
CPU Memory
SELECT NAME FROM CUSTOMERS WHERE STATUS ='ANGRY'
Filtering should be Applied Here
Network I/O Controller Disk
Exadata: from Beginner to Advanced in 3 Hours
10
The Magic #1 CPU
iDB
Memory Network I/O Controller Disk Exadata: from Beginner to Advanced in 3 Hours
The communication between CPU and Disk carries the information on the query – columns and predicates. This occurs as a result of a special protocol called iDB. 11
Magic #2 Storage Cell Server • •
iDB
•
Disk1
Disk2
Cells are Sun Blades Run Oracle Enterprise Linux Software called Exadata Storage Server (ESS) which understands iDB
Disk3
Exadata: from Beginner to Advanced in 3 Hours
12
Magic #3 Storage Indexes Storage Indexes store in memory of the Cell Server the areas on the disk and the MIN/MAX value of the column and whether NULL exists. They eliminate disk I/O. SELECT … FROM TABLE WHERE COL1 = 1
Storage Index
Disk1
Disk2
Disk3
MIN = 3
MIN = 4
MIN = 1
MIN = 3
MAX = 5
MAX = 5
MAX = 2
MAX = 5
Disk4 Exadata: from Beginner to Advanced in 3 Hours 13
Checking Storage Index Use select name, value/1024/1024 as stat_value from v$mystat s, v$statname n where s.statistic# = n.statistic# and n.name in ( 'cell physical IO bytes saved by storage index', 'cell physical IO interconnect bytes returned by smart scan’) Output STAT_NAME STAT_VALUE ---------- ---------SI Savings 5120.45 Smart Scan 1034.00 Exadata: from Beginner to Advanced in 3 Hours
14
Checking Offloading of an SQL select sql_id, child_number child#, plan_hash_value plan_hash, executions execs, (elapsed_time/1000000)/decode(nvl(executions,0),0,1,executions)/ decode(px_servers_executions,0,1,px_servers_executions/decode(nvl(executions,0),0,1, executions)) avg_elapsed_time_in_secs, px_servers_executions/decode(nvl(executions,0),0,1,executions) avg_par_deg, decode(io_cell_offload_eligible_bytes,0,'No','Yes') Offloaded, decode(io_cell_offload_eligible_bytes,0,0,100*(io_cell_offload_eligible_bytesio_interconnect_bytes) /decode(io_cell_offload_eligible_bytes,0,1,io_cell_offload_eligible_bytes)) "%age IO Saved", buffer_gets/decode(nvl(executions,0),0,1,executions) avg_lio from v$sql where sql_text like %'
Exadata: from Beginner to Advanced in 3 Hours
15
Why Not? • Pre-requisite for Smart Scan Direct Path Full Table or Full Index Scan > 0 Predicates Disabling Smart Scans cell_offload_processing = Simple Comparison Operators false; _kcfis_storageidx_disabled = • Other Reasons – Cell is not offload capable true; – – – –
• The diskgroup attribute cell.smart_scan_capable set to FALSE;
– Not on clustered tables, IOTs, etc. Exadata: from Beginner to Advanced in 3 Hours
16
Magic #4 Flash Cache These are flash cards presented as disks; not memory to the Storage Cells. They are similar to SAN cache; but Oracle controls what goes on there and how long it stays.
datafile1 datafile2
Storage Exadata: from Beginner to Advanced in 3 Hours
17
Magic #5 Process Offloading • Bloom Filters • Functions Offloading – Get the functions that can be offloaded • V$SQLFN_METADATA
• Decompression – (Compression handled by Compute Nodes)
• Virtual Columns
Exadata: from Beginner to Advanced in 3 Hours
18
Components CPU
Database Node (Sun Blade. OEL)
Memory
Oracle 11gR2 RAC
Network
InfiniBand Switch
I/O Controller
Storage Cell
Disk
Exadata Storage Server Disks, Flash Exadata: from Beginner to Advanced in 3 Hours
19
Put Together: One Full Rack RAC Cluster
Database Database Node 1 Database Node 1 Database Node 1 Node 1 InfiniBand Switch
Cell 1 Cell 1 Cell 1 Cell 1
Database Node 8
Network Switch
Clients connect to the database nodes.
Cell 14
Exadata: from Beginner to Advanced in 3 Hours
20
How it Looks X2-2 Qtr
X2-2 Half
X2-2 Full
X2-8 Full
Number of Compute Nodes Total Compute Node Processor Cores Total Compute Node Memory Number of Storage Servers Number of SAS Disks in Storage
2
4
8
2
24
48
96
160
196 GB
384 GB
768 GB
4 TB
3
7
14
14
36
84
168
168
Storage Capacity - HP
21.6 TB
50.4 TB
100.8 TB
100.8 TB
Storage Capacity - HC
108 TB
252 TB
504 TB
504 TB
Number of InfiniBand Switches
2
3
3
3
Source: upcoming book Exadata Recipes by Clarke from Apress
Exadata: from Beginner to Advanced in 3 Hours
21
Disk Layout •
Compute Nodes
• Storage Cell
• • •
Disks (hard and flash) are connected to the cells. The disks are partitioned at the cell Some partitions are presented as filesystems The rest are used for ASM diskgroups All these disks/partitions are presented to the compute nodes
Exadata: from Beginner to Advanced in 3 Hours
22
Node
Disk Presentation
filesystem
Cell
filesystem
Exadata: from Beginner to Advanced in 3 Hours
23
Command Components Compute Nodes
Storage Cell
Linux Commands – vmstat, mpstat, fdisk, etc. ASM Commands – SQL*Plus, ASMCMD, ASMCA Database Commands – startup, alter database, etc. Clusterware Commands – CRSCTL, SRVCTL, etc. Linux Commands – vmstat, mpstat, fdisk, etc. CellCLI – command line tool to manage the Cell 5-part Linux Commands article series http://bit.ly/k4mKQS 4-part Exadata Command Reference article series http://bit.ly/lljFl0
Exadata: from Beginner to Advanced in 3 Hours
24
Administration Skills Skill System Administrator Storage Administrator Network Administrator Database Administrator Cell Administration
Needed 15% 0% 5% 60% 20% DBA Sys Admin Network Admin
Exadata: from Beginner to Advanced in 3 Hours
Cell Admin 25
One Cluster?
One Cluster
QA1
QA2
QA3
Prod1
Prod2
Prod3
Dev1
Int1
QA1
QA2
Prod4
Prod1
Prod2
Prod3
Dev1
Int1
QA1
QA2
QA3
Prod1
Prod2
Prod3
Dev1
Int1 Dev2
Exadata: from Beginner to Advanced in 3 Hours
26
Many Clusters?
QA Cluster
QA1
QA2
QA Cluster
Prod Cluster
QA3
Prod1
Prod2
Prod Cluster Exadata: from Beginner to Advanced in 3 Hours
Prod3
Dev
Int
Dev1
Int1
Dev
Int 27
Disk Failures Datafile block1
Cell 1
Cell 2
block1
block1
Exadata: from Beginner to Advanced in 3 Hours
28
Disk Failures Datafile block1
Cell 1 block1
Cell 2
Cell 3
block1
block1
High Redundancy Exadata: from Beginner to Advanced in 3 Hours
29
Playing Nice • Database Resource Manager • I/O Resource Manager • Cell Fencing QA
Prod
Compute Nodes
Storage Cells
Exadata: from Beginner to Advanced in 3 Hours
30
Divide and Conquer Database
DBA
Machine
System Admin
Network
Network Admin
Storage
??
DBA Sys Admin Network Admin Cell Admin
Exadata: from Beginner to Advanced in 3 Hours
31
Combined Skills Database Machine Administrator (DMA)
Database Machine New Role Network Storage
DBA Sys Admin Network Admin Cell Admin Exadata: from Beginner to Advanced in 3 Hours
32
Other Questions Q: Do clients have to connect using Infiniband? A: No; Ethernet is also available
Q: How do you back it up? A: Normal RMAN Backup, just like an Oracle Database
Q: How do you create DR? A: Data Guard is the only solution
Q: Can I install any other software? A: Nothing on Cells. On nodes – yes
Q: How do I monitor it? A: Enterprise Manager, CellCLI, SQL Commands Exadata: from Beginner to Advanced in 3 Hours
33
Backup and DR • No SAN connectivity • Only NAS
Exadata
– Infiniband – Tape , Disk Pool
• DR – – – – –
Infiniband
No Storage Level Replication Backup Only Data Guard Supplemental Logging Device Force Logging http://www.oracle.com/technetwork/database/features/availa bility/maa-wp-dr-dbm-130065.pdf
• Golden Gate Exadata: from Beginner to Advanced in 3 Hours
34
ETL and Reporting
OLTP
Exadata Golden Gate Infiniband
Informatica
MicroStrategy Exalytics
Exadata: from Beginner to Advanced in 3 Hours
35
Overall Activities • Physical Aspects – Delivery, power, network components, etc.
• • • •
Layout Planning Installation and Configuration Data Migration Administration – Who manages it – Backup and Disaster Recovery
• Application Development Exadata: from Beginner to Advanced in 3 Hours
36
Power Delivery Units • Over or Under the unit • Power Requirement – Single-Phase Low Voltage Americas / Japan / Taiwan) – Single-Phase High Voltage (EMEA & APAC (excluding Japan / Taiwan) – Three-Phase Low Voltage (Americas / Japan / Taiwan) – Three-Phase High Voltage (EMEA & APAC (excluding Japan / Taiwan)
Exadata: from Beginner to Advanced in 3 Hours
37
Network Ports • NET0 – Admin Interface
• NET1, NET2 – Network Access to Nodes
• NET3 – Backup Network
• IB – Infiniband Network – IP Addr: Qtr Rack: 5; Half Rack: 11; Full rack: 22 Exadata: from Beginner to Advanced in 3 Hours
38
Configuration Worksheet Pre-delivery Survey Generate config files Run checkip.sh Power on and validate components Configure KVM Configure IB Configure Cisco Switch Exadata: from Beginner to Advanced in 3 Hours
Oracle HW
1. 2. 3. 4. 5. 6. 7. 8.
ACS
Installation Activities
39
Exadata: from Beginner to Advanced in 3 Hours
ACS
9. Configure IP to PDUs 10. Validate Storage Cells 11. Validate Compute Nodes 12. Config files from USB 13. Firstboot and applyconfig.sh 14. Stage Oracle Software on Node 1 15. Run OneCommand
Oracle HW
Installation, contd.
40
Summary • Exadata is an Oracle Database running 11.2 • The storage cells have added intelligence about data placement • The compute nodes run Oracle DB and Grid Infra • Nodes communicate with Cells using iDB which can send more information on the query • Smart Scan, when possible, reduces I/O at cells even for full table scans • Cell is controlled by CellCLI commands • DMA skills = 60% RAC DBA + 15% Linux + 20% CellCLI + 5% miscellaneous Exadata: from Beginner to Advanced in 3 Hours
41
Resources • My Articles – 5-part Linux Commands article series http://bit.ly/k4mKQS – 4-part Exadata Reference article series http://bit.ly/lljFl0
• OTN Page on Exadata – http://www.oracle.com/technetwork/database/exadata/index. html
• Tutorials – http://www.oracle.com/technetwork/tutorials/index.html
• OTN Exadata Forum – https://forums.oracle.com/forums/forum.jspa?forumID=829
• Exadata SIG – http://www.linkedin.com/groups?home=&gid=918317 Exadata: from Beginner to Advanced in 3 Hours
42
Thank You! My Blog: arup.blogspot.com My Tweeter: arupnanda
Exadata: from Beginner to Advanced in 3 Hours
43
