您好,欢迎光临! 请 |

Oracle 10g RAC 配置物理dataguard系列2:备库安装、升级ora

作者: | 分类:Dataguard, Oracle, RAC | Tag:, | 评论:2 
字号:T|T

在上一篇文章Oracle 10g RAC 配置物理dataguard系列1:RAC主库信息概览、备库准备工作中,我们了解了双节点RAC主库的基本信息,并且成功创建了一台将来用作物理备库的机器。在本篇中,我们将完成以下任务:

  • 在物理备库机器上安装oracle 10g 数据库软件;
  • 在物理备库机器上升级oracle 10g 软件到10.2.0.5.0;
  • 在物理备库机器上配置ASM存储 ,创建ASM实例;

一     安装oracle软件,在一台干净的Linux服务器上安装一套oracle 10g数据库软件,需要完成以下任务:

        1 创建oracle用户、用户组,参照以下shell脚本完成:

[root@ora10grac-dg Pre10g]# pwd
/root/Pre10g
[root@ora10grac-dg Pre10g]# ll
total 24
-rw-r--r-- 1 root root 2165 Feb 14 14:49 1preuser.sh
-rw-r--r-- 1 root root  510 Feb 14 14:47 2predir.sh
-rw-r--r-- 1 root root  801 Feb 14 14:47 3prelimits.sh
-rw-r--r-- 1 root root  541 Feb 14 14:47 4prelogin.sh
-rw-r--r-- 1 root root  652 Feb 14 14:47 5preprofile.sh
-rw-r--r-- 1 root root 2300 Feb 14 14:47 6presysctl.sh
[root@ora10grac-dg Pre10g]# chmod u+x *.sh
[root@ora10grac-dg Pre10g]# ll
total 24
-rwxr--r-- 1 root root 2165 Feb 14 14:49 1preuser.sh
-rwxr--r-- 1 root root  510 Feb 14 14:47 2predir.sh
-rwxr--r-- 1 root root  801 Feb 14 14:47 3prelimits.sh
-rwxr--r-- 1 root root  541 Feb 14 14:47 4prelogin.sh
-rwxr--r-- 1 root root  652 Feb 14 14:47 5preprofile.sh
-rwxr--r-- 1 root root 2300 Feb 14 14:47 6presysctl.sh
[root@ora10grac-dg Pre10g]# cat 1preuser.sh 
#!/bin/bash
#Purpose:Create two groups named 'oinstall' and 'dba', plus a user named 'oracle'.Also setting the Environment
#variable for oracle user.
# 
#Usage:Log on as the superuser('root'),and then execute the command:#./1preuser.sh
#Author:Asher Huang

echo "Now create two groups named 'oinstall' and 'dba', plus a user named 'oracle'.Also setting the Environment"
groupadd oinstall 
groupadd dba
useradd -g oinstall -G dba -c "Oracle software owner" -d  /home/oracle   oracle 
echo "oracle" | passwd --stdin oracle
echo "export ORACLE_BASE=/u01/app" >> /home/oracle/.bash_profile
echo 'export ORACLE_HOME=$ORACLE_BASE/oracle' >> /home/oracle/.bash_profile
echo "export ORACLE_SID=pridb"  >> /home/oracle/.bash_profile
echo 'export LD_LIBRARY_PATH=$ORACLE_HOME/lib'  >> /home/oracle/.bash_profile
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib:/usr/X11R6/lib'  >> /home/oracle/.bash_profile
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/jdk/jre/lib/i386'  >> /home/oracle/.bash_profile
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/jdk/jre/lib/i386/server'  >> /home/oracle/.bash_profile
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/rdbms/lib'  >> /home/oracle/.bash_profile
echo 'export CLASS_PATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib'  >> /home/oracle/.bash_profile
echo 'export CLASS_PATH=$CLASS_PATH:$ORACLE_HOME/rdbms/jlib'  >> /home/oracle/.bash_profile
echo 'export CLASS_PATH=$CLASS_PATH:$ORACLE_HOME/network/jlib'  >> /home/oracle/.bash_profile
echo 'export TNS_ADMIN=$ORACLE_HOME/network/admin'  >> /home/oracle/.bash_profile
echo "export NLS_LANG=american_america.AL32UTF8" >> /home/oracle/.bash_profile
echo "export ORACLE_TERM=xterm" >> /home/oracle/.bash_profile
echo "export EDITOR=vi" >> /home/oracle/.bash_profile
echo 'export PATH=$ORACLE_HOME/bin:$PATH' >> /home/oracle/.bash_profile
echo "export LANG=en_US" >> /home/oracle/.bash_profile
echo "export NLS_DATE_FORMAT='yyyy/mm/dd hh24:mi:ss'" >> /home/oracle/.bash_profile

echo "The two groups named 'oinstall' and 'dba' plus the oracle user has been created."
echo "The Environment variable for oracle user also has been setted sussessfully."
[root@ora10grac-dg Pre10g]# 

        2 创建相关目录,其中/u01/app用做Oracle Base 目录,/u01/app/oracle用作Oracle Home目录,具体参照下述脚本:

[root@ora10grac-dg Pre10g]# cat 2predir.sh 
#!/bin/bash
#Purpose:Create the necessary directory for oracle user and change the authention to oracle user.
#Usage:Log on as the superuser('root'),and then execute the command:#./2predir.sh
#Author:Asher Huang

echo "Now create the necessary directory for oracle user and change the authention to oracle user..."
mkdir -p /u01/app/oracle
chown -R oracle:oinstall /u01
chown oracle:oinstall /home/oracle
echo "The necessary directory for oracle user and change the authention to oracle user has been finished"[root@ora10grac-dg Pre10g]#

        3 配置oracle用户的SHELL限制,我们通过下述脚本来修改/etc/security/limits.conf配置文件:

[root@ora10grac-dg Pre10g]# cat 3prelimits.sh 
#!/bin/bash
#Purpose:Change the /etc/security/limits.conf. 
#        "Configure the SHELL limits on oracle user."
#        配置oracle用户的SHELL限制
#  nofile ----可打开的文件描述符的最大数
#  nproc  ----单个用户可用的最大进程数量
#Usage:Log on as the superuser('root'),and then execute the command:#./3prelimits.sh
#Author:Asher Huang


echo "Now modify the /etc/security/limits.conf,but backup it named /etc/security/limits.conf.bak before"
cp /etc/security/limits.conf /etc/security/limits.conf.bak
echo "oracle soft nproc 2047" >>/etc/security/limits.conf
echo "oracle hard nproc 16384" >>/etc/security/limits.conf
echo "oracle soft nofile 1024" >>/etc/security/limits.conf
echo "oracle hard nofile 65536" >>/etc/security/limits.conf
echo "Modifing the /etc/security/limits.conf has been succeed."[root@ora10grac-dg Pre10g]# 

        4 配置SHELL对oracle用户的限制生效,需要修改/etc/pam.d/login文件,脚本如下:

[root@ora10grac-dg Pre10g]# cat 4prelogin.sh 
#!/bin/bash
#Purpose:Modify the /etc/pam.d/login.
# 为使第3步骤中配置SHELL对oracle用户的限制生效,需要修改/etc/pam.d/login文件.
#Usage:Log on as the superuser('root'),and then execute the command:#./4prelimits.sh
#Author:Asher Huang

echo "Now modify the /etc/pam.d/login,but with a backup named /etc/pam.d/login.bak"
cp /etc/pam.d/login /etc/pam.d/login.bak
echo "session required /lib/security/pam_limits.so" >>/etc/pam.d/login
echo "session required pam_limits.so" >>/etc/pam.d/login
echo "Modifing the /etc/pam.d/login has been succeed."

        5 配置系统相关的环境变量,修改/etc/profile,脚本:

[root@ora10grac-dg Pre10g]# cat 5preprofile.sh 
#!/bin/bash
#Purpose:Modify the /etc/profile.配置系统相关的环境变量
#Usage:Log on as the superuser('root'),and then execute the command:#./5preprofile.sh
#Author:Asher Huang

echo "Now modify the  /etc/profile,but with a backup named  /etc/profile.bak"
cp /etc/profile /etc/profile.bak
echo 'if [ $USER = "oracle" ]; then' >>  /etc/profile
echo 'if [ $SHELL = "/bin/ksh" ]; then' >> /etc/profile
echo 'ulimit -p 16384' >> /etc/profile
echo 'ulimit -n 65536' >> /etc/profile
echo 'else' >> /etc/profile
echo 'ulimit -u 16384 -n 65536' >> /etc/profile
echo 'fi' >> /etc/profile
echo 'fi' >> /etc/profile
echo "Modifing the /etc/profile has been succeed."

        6 修改系统内核参数,修改/etc/sysctl.conf配置文件,脚本:

[root@ora10grac-dg Pre10g]# cat 6presysctl.sh 
#!/bin/bash
#Purpose:Modify the /etc/sysctl.conf.
# 修改系统内核参数,若配置有误,在安装时有先决条件检查,即会报错!!!
#Usage:Log on as the superuser('root'),and then execute the command:#./6presysctl.sh
#Author:Asher Huang

echo "Now modify the /etc/sysctl.conf,but with a backup named /etc/sysctl.bak"
cp /etc/sysctl.conf /etc/sysctl.conf.bak
echo "fs.file-max = 65536" >> /etc/sysctl.conf
echo "kernel.sem = 250 32000 100 128" >> /etc/sysctl.conf

echo "kernel.shmall = 2097152" >> /etc/sysctl.conf
echo "kernel.shmmni = 4096" >> /etc/sysctl.conf
echo "kernel.shmmax = 2147483648" >> /etc/sysctl.conf

echo "net.core.rmem_default=1048576" >> /etc/sysctl.conf
echo "net.core.rmem_max=262144" >> /etc/sysctl.conf
echo "net.core.wmem_default=262144" >> /etc/sysctl.conf
echo "net.core.wmem_max=262144" >> /etc/sysctl.conf

echo "net.ipv4.ip_local_port_range = 1024 65000" >> /etc/sysctl.conf
echo "Modifing the /etc/sysctl.conf has been succeed."
echo "Now make the changes take effect....."

## fs.file-max = 65536 表示文件句柄的最大数量。文件句柄设置表示在linux系统中可以打开的文件数量 

## kernel.sem = 250 32000 100 128  sem 4个参数依次是
##SEMMSL:每个用户拥有信号量最大数;
##SEMMNS:系统信号量最大数;
##SEMOPM:每次semopm系统调用操作数;
##SEMMNI:系统信号量集数最大数。这4个参数为固定内容大小。

## kernel.shmall = 2097152 该参数表示系统一次可以使用的共享内存总量(以页为单位,)。缺省值就是2097152,通常不需要修改。
## 可用共享内存的总量,单位页,在32位系统上一页=4k,也就是4096字节。按照这个设置2097152*4k/1024/1024 = 8G, 就是说可用共享内存一共 8G
## kernel.shmmax = 2147483648 该参数定义了共享内存段的最大尺寸(以字节为单位)。缺省为32M,对于oracle来说,该缺省值太低了,通常将其设置为2G
## 单个共享内存段的最大数,2147483648/1024/1024=2048M=2G
## kernel.shmmni = 4096 这个内核参数用于设置系统范围内单个共享内存段的最小值。该参数的默认值是 4096 。通常不需要更改。


## 设置UDP send/receive buffers high enough ,通常在RAC环境下,私网最好是千兆网卡,协议为UDP。
## net.core.rmem_default=262144   //  默认的接收窗口大小 
## net.core.rmem_max=262144       //  接收窗口的最大大小 
## net.core.wmem_default=262144   //  默认的发送窗口大小 
## net.core.wmem_max=262144       //  发送窗口的最大大小


## net.ipv4.ip_local_port_range = 1024 65000 //表示端口范围


#使配置生效,执行下述命令:
sysctl -p

        7 将上述脚本编写无误后,以root用户依次执行:

[root@ora10grac-dg Pre10g]# ./1preuser.sh 
Now create two groups named 'oinstall' and 'dba', plus a user named 'oracle'.Also setting the Environment
Changing password for user oracle.
passwd: all authentication tokens updated successfully.
The two groups named 'oinstall' and 'dba' plus the oracle user has been created.
The Environment variable for oracle user also has been setted sussessfully.
[root@ora10grac-dg Pre10g]# ./2predir.sh 
Now create the necessary directory for oracle user and change the authention to oracle user...
The necessary directory for oracle user and change the authention to oracle user has been finished
[root@ora10grac-dg Pre10g]# ./3prelimits.sh 
Now modify the /etc/security/limits.conf,but backup it named /etc/security/limits.conf.bak before
Modifing the /etc/security/limits.conf has been succeed.
[root@ora10grac-dg Pre10g]# ./4prelogin.sh 
Now modify the /etc/pam.d/login,but with a backup named /etc/pam.d/login.bak
Modifing the /etc/pam.d/login has been succeed.
[root@ora10grac-dg Pre10g]# ./5preprofile.sh 
Now modify the  /etc/profile,but with a backup named  /etc/profile.bak
Modifing the /etc/profile has been succeed.
[root@ora10grac-dg Pre10g]# ./6presysctl.sh 
Now modify the /etc/sysctl.conf,but with a backup named /etc/sysctl.bak
Modifing the /etc/sysctl.conf has been succeed.
Now make the changes take effect.....
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 65536
kernel.sem = 250 32000 100 128
kernel.shmall = 2097152
kernel.shmmni = 4096
kernel.shmmax = 2147483648
net.core.rmem_default = 1048576
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.ipv4.ip_local_port_range = 1024 65000
[root@ora10grac-dg Pre10g]#  

        8 以oracle用户登录系统,并确认环境变量生效:

[root@ora10grac-dg Pre10g]# id oracle
uid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba)
[root@ora10grac-dg Pre10g]# su - oracle
[oracle@ora10grac-dg ~]$ pwd
/home/oracle
[oracle@ora10grac-dg ~]$ env | grep ORA
ORACLE_SID=pridb
ORACLE_BASE=/u01/app
ORACLE_TERM=xterm
ORACLE_HOME=/u01/app/oracle
[oracle@ora10grac-dg ~]$ 

        9 挂载oracle软件安装介质,并以oracle用户登录图形界面,开始执行软件安装:

          ①  挂载安装介质,这里oracle 10g软件的安装包、升级包、集群件已经刻录好成一个ISO文件,所以只需挂载光盘:

[root@ora10grac-dg ~]# mount /dev/cdrom /mnt/
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@ora10grac-dg ~]# cd /mnt/
[root@ora10grac-dg mnt]# ll
total 6
dr-xr-xr-x 1 root root 2048 Jul 27  2010 10201_clusterware_linux_x86_64
dr-xr-xr-x 1 root root 2048 Jul 27  2010 10201_database_linux_x86_64
dr-xr-xr-x 1 root root 2048 Jul 27  2010 p8202632_10205_Linux-x86-64
[root@ora10grac-dg mnt]# cd 10201_database_linux_x86_64/ && ll
total 25
dr-xr-xr-x 1 root root  2048 Jul 27  2010 doc
dr-xr-xr-x 1 root root  2048 Jul 27  2010 install
dr-xr-xr-x 1 root root  2048 Jul 27  2010 response
-r-xr-xr-x 1 root root  1326 Oct 23  2005 runInstaller
dr-xr-xr-x 1 root root 12288 Jul 27  2010 stage
-r-xr-xr-x 1 root root  5416 Oct 20  2005 welcome.html
[root@ora10grac-dg 10201_database_linux_x86_64]# 

          ② Oracle用户登录图形界面,执行安装:

           ③   发现报错,原因是Oracle 10g 软件不支持Red Hat 5版本的操作系统,Oracle 10g在2004年就发布了,而Red Hat 5则在2007年发布。修改/etc/redhat-release文件,使之骗过Oracle:

[root@ora10grac-dg ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
[root@ora10grac-dg ~]# cp /etc/redhat-release /etc/redhat-release.bak
[root@ora10grac-dg ~]# vi /etc/redhat-release
Red Hat Enterprise Linux Server release 4 (Tikanga)
[root@ora10grac-dg ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 4 (Tikanga)
[root@ora10grac-dg ~]# 

          ④     修改之后,重新执行OUI图形界面安装,发现这次报错如下:

[oracle@ora10grac-dg ~]$ /mnt/10201_database_linux_x86_64/runInstaller
Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be redhat-3, SuSE-9, redhat-4, UnitedLinux-1.0, asianux-1 or asianux-2
                                      Passed


All installer requirements met.

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-02-14_03-25-08PM. Please wait ...[oracle@ora10grac-dg ~]$ Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/OraInstall2012-02-14_03-25-08PM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(Unknown Source)
        at java.lang.ClassLoader.loadLibrary(Unknown Source)
        at java.lang.Runtime.loadLibrary0(Unknown Source)
        at java.lang.System.loadLibrary(Unknown Source)
        at sun.security.action.LoadLibraryAction.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.awt.NativeLibLoader.loadLibraries(Unknown Source)
        at sun.awt.DebugHelper.(Unknown Source)
        at java.awt.Component.(Unknown Source)
[oracle@ora10grac-dg ~]$ 

          ⑤      原因是缺少libXp-1.0.0-8.1.el5.i386.rpm及libXp-1.0.0-8.1.el5.x86_64.rpm这两个RPM软件包,接下来,卸载oracle安装介质的光盘,挂载操作系统安装光盘,到操作系统安装光盘里找到这两个RPM包,并执行安装:

[root@ora10grac-dg ~]# umount /mnt/
[root@ora10grac-dg ~]# mount /dev/cdrom /mnt/
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@ora10grac-dg ~]# cd /mnt/Server/
[root@ora10grac-dg Server]# ll libXp*
-rw-r--r-- 2 oracle root 22852 Jan 22  2009 libXp-1.0.0-8.1.el5.i386.rpm
-rw-r--r-- 2 oracle root 23561 Jan 22  2009 libXp-1.0.0-8.1.el5.x86_64.rpm
-rw-r--r-- 2 oracle root 15103 Jan 22  2009 libXp-devel-1.0.0-8.1.el5.i386.rpm
-rw-r--r-- 2 oracle root 15072 Jan 22  2009 libXp-devel-1.0.0-8.1.el5.x86_64.rpm
-rw-r--r-- 2 oracle root 46410 Jan 22  2009 libXpm-3.5.5-3.i386.rpm
-rw-r--r-- 2 oracle root 45336 Jan 22  2009 libXpm-3.5.5-3.x86_64.rpm
-rw-r--r-- 2 oracle root 31307 Jan 22  2009 libXpm-devel-3.5.5-3.i386.rpm
-rw-r--r-- 2 oracle root 32469 Jan 22  2009 libXpm-devel-3.5.5-3.x86_64.rpm
[root@ora10grac-dg Server]# rpm -ivh libXp-1.0.0-8.1.el5.i386.rpm 
warning: libXp-1.0.0-8.1.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:libXp                  ########################################### [100%]
[root@ora10grac-dg Server]# rpm -ivh libXp-1.0.0-8.1.el5.x86_64.rpm 
warning: libXp-1.0.0-8.1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:libXp                  ########################################### [100%]
[root@ora10grac-dg Server]# cd 
[root@ora10grac-dg ~]# umount /mnt/
[root@ora10grac-dg ~]# 

            ⑥    接下来,重新挂载Oracle安装介质光盘,重新执行OUI安装,进入熟悉的OUI安装界面,选择高级安装,单击Next,一路走下去:

                    单击Next,进入到安装前的与检查页面,提示交换分区不够大的警告,还有一个网络配置的信息,说是建议采用DHCP,这里直接忽略:

          ⑧    Next 进入下一步,注意选择只安装软件,不要建库,因为我们是要配置物理dataguard。

             ⑨       最后,以root用户执行下述两个脚本,完成oracle软件的安装:

[root@ora10grac-dg ~]# /u01/app/oraInventory/orainstRoot.sh 
Changing permissions of /u01/app/oraInventory to 770.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete  
[root@ora10grac-dg ~]# /u01/app/oracle/root.sh
Running Oracle10 root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
[root@ora10grac-dg ~]# 

         ⑩         以Oracle用户执行SQL*PLUS命令行工具,做简单验证:

[root@ora10grac-dg ~]# su - oracle
[oracle@ora10grac-dg ~]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Feb 14 16:53:01 2012
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> 

至此,Oracle 10g数据库软件已经安装完成!!!

二      oracle软件安装完成之后,接下来,开始升级Oracle软件到10.2.0.5.0版本:

       ①    同样,10.2.0.5.0的升级包也在安装介质的光盘上,如果没有的话,需要自行到oracle的官方网站下载。依然以oracle用户在图形界面下执行下述命令,进入OUI的图形界面进行升级:

[oracle@ora10grac-dg ~]$ /mnt/p8202632_10205_Linux-x86-64/Disk1/runInstaller 
Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be redhat-3, SuSE-9, SuSE-10, redhat-4, redhat-5, redhat-6, UnitedLinux-1.0, asianux-1, asianux-2, asianux-3, enterprise-4, enterprise-5 or SuSE-11
                                      Passed


All installer requirements met.

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2012-02-14_04-59-35PM. Please wait ...[oracle@ora10grac-dg ~]$ Oracle Universal Installer, Version 10.2.0.5.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

     

②   Next 进入下一步:

       ③   继续Next:

④    Next,进入下一步:

⑤   发现,此时提示内核的fs.file-max = 65536参数太小,需要调整为101365。以及net.ipv4.ip_local_port_range = 1024 65000需要调整为net.ipv4.ip_local_port_range = 9000 65500。依据错误提示信息,对/etc/sysctl.conf配置文件重新进行修改。修改后,单击Retry,此时不再提示该警告,转而提示swap过小,可直接忽略:

[root@ora10grac-dg ~]# tail -10 /etc/sysctl.conf
fs.file-max = 101365
kernel.sem = 250 32000 100 128
kernel.shmall = 2097152
kernel.shmmni = 4096
kernel.shmmax = 2147483648
net.core.rmem_default=1048576
net.core.rmem_max= 1048576
net.core.wmem_default=262144
net.core.wmem_max=262144
net.ipv4.ip_local_port_range = 9000 65500
[root@ora10grac-dg ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 101365
kernel.sem = 250 32000 100 128
kernel.shmall = 2097152
kernel.shmmni = 4096
kernel.shmmax = 2147483648
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.ipv4.ip_local_port_range = 9000 65500
[root@ora10grac-dg ~]# 

⑥    单击,Install,进入安装升级包:

 ⑦       最后,执行提示脚本/u01/app/oracle/root.sh,完成oracle软件的升级!

 

[root@ora10grac-dg ~]# /u01/app/oracle/root.sh
Running Oracle 10g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
[root@ora10grac-dg ~]# 

⑧    当然,再次以oracle用户在命令行执行SQL*PLUS命令,确认数据库软件已经升级到10.2.0.5.0版本:

[root@ora10grac-dg ~]# su - oracle
[oracle@ora10grac-dg ~]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.5.0 - Production on Tue Feb 14 17:31:12 2012

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

SQL> 

⑨   最后,别忘记了配置监听程序。可以使用Oracle提供的Net Configuration Assisstant、Net Manager图形化界面工具进行配置,也可以直接到$ORACLE_HOME/network/admin/目录下直接编辑一个listener.ora的配置文件。这里以netca配置:

配置完成后的信息如下:

[oracle@ora10grac-dg ~]$ cat /u01/app/oracle/network/admin/listener.ora 
# listener.ora Network Configuration File: /u01/app/oracle/network/admin/listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracle)
      (PROGRAM = extproc)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ora10grac-dg)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
    )
  )

[oracle@ora10grac-dg ~]$ lsnrctl status

LSNRCTL for Linux: Version 10.2.0.5.0 - Production on 15-FEB-2012 09:17:04

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ora10grac-dg)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.2.0.5.0 - Production
Start Date                15-FEB-2012 09:16:03
Uptime                    0 days 0 hr. 1 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/network/admin/listener.ora
Listener Log File         /u01/app/oracle/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ora10grac-dg)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@ora10grac-dg ~]$ 

三 接下去的工作便是配置ASM存储,创建ASM实例:

      ① 关闭物理备库机器,添加新的硬盘,该硬盘将来用作ASM磁盘,用于数据库的存储。我这里添加两块硬盘/dev/sdb将来用作ASM存储,/dev/sdc将来用作备份用。添加硬盘后的信息如下:

[root@ora10grac-dg ~]# df -Th
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    7.8G  4.8G  2.6G  66% /
tmpfs        tmpfs    1.1G     0  1.1G   0% /dev/shm
[root@ora10grac-dg ~]# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1044     8385898+  83  Linux
/dev/sda2            1045        1305     2096482+  82  Linux swap / Solaris

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn't contain a valid partition table
[root@ora10grac-dg ~]# 

    接下来,分别格式化/dev/sdb、/dev/sdc:

[root@ora10grac-dg ~]# fdisk /dev/sdb 
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305): 
Using default value 1305

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@ora10grac-dg ~]# fdisk /dev/sdc
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 1044.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1044, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1044, default 1044): 
Using default value 1044

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@ora10grac-dg ~]# partprobe 
Warning: Unable to open /dev/hdc read-write (Read-only file system).  /dev/hdc has been opened read-only.
[root@ora10grac-dg ~]# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1044     8385898+  83  Linux
/dev/sda2            1045        1305     2096482+  82  Linux swap / Solaris

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1305    10482381   83  Linux

Disk /dev/sdc: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        1044     8385898+  83  Linux
[root@ora10grac-dg ~]# 

 在这里,顺便给/dev/sdc1做ext3的文件系统,然后挂载到/oracle_backup下,并更改权限,步骤如下:       

[root@ora10grac-dg ~]# mkdir /oracle_backup  
[root@ora10grac-dg ~]# mkfs.ext3 /dev/sdc1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1048576 inodes, 2096474 blocks
104823 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2147483648
64 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@ora10grac-dg ~]# mount -t ext3 /dev/sdc1 /oracle_backup/
[root@ora10grac-dg ~]# df -Th
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    7.8G  4.8G  2.6G  66% /
tmpfs        tmpfs    1.1G     0  1.1G   0% /dev/shm
/dev/sdc1     ext3    7.9G  147M  7.4G   2% /oracle_backup
[root@ora10grac-dg ~]# chown -R oracle:oinstall /oracle_backup/
[root@ora10grac-dg ~]# cd / && ll
total 184
drwxr-xr-x   2 root   root      4096 Feb 14 12:26 bin
drwxr-xr-x   3 root   root      4096 Feb 14 11:07 boot
drwxr-xr-x  10 root   root      3800 Feb 15 09:57 dev
drwxr-xr-x  89 root   root     12288 Feb 15 10:03 etc
drwxr-xr-x   3 root   root      4096 Feb 14 14:59 home
drwxr-xr-x  11 root   root      4096 Feb 14 12:26 lib
drwxr-xr-x   7 root   root     12288 Feb 14 12:26 lib64
drwx------   2 root   root     16384 Feb 14 11:00 lost+found
drwxr-xr-x   2 root   root      4096 Feb 15 09:38 media
drwxr-xr-x   2 root   root         0 Feb 15 09:38 misc
drwxr-xr-x   2 root   root      4096 Feb 17  2010 mnt
drwxr-xr-x   2 root   root         0 Feb 15 09:38 net
drwxr-xr-x   3 root   root      4096 Feb 14 16:43 opt
drwxr-xr-x   3 oracle oinstall  4096 Feb 15 10:03 oracle_backup
dr-xr-xr-x 120 root   root         0 Feb 15 09:37 proc
drwxr-x---  16 root   root      4096 Feb 14 17:46 root
drwxr-xr-x   2 root   root     12288 Feb 14 12:26 sbin
drwxr-xr-x   2 root   root      4096 Feb 14 11:01 selinux
drwxr-xr-x   2 root   root      4096 Feb 17  2010 srv
drwxr-xr-x  11 root   root         0 Feb 15 09:37 sys
drwxr-xr-x   3 root   root      4096 Feb 14 11:06 tftpboot
drwxrwxrwt  17 root   root      4096 Feb 15 09:51 tmp
drwxr-xr-x   3 oracle oinstall  4096 Feb 14 14:59 u01
drwxr-xr-x  15 root   root      4096 Feb 14 11:05 usr
drwxr-xr-x  22 root   root      4096 Feb 14 11:10 var
[root@ora10grac-dg /]# 

        ② 配置ASM,在这里用裸设备做,不采用oracle 的asmlib进行配置(虽然我个人以前一直使用),所以在这里也就不需要安装asmlib软件包

             首先,获取/dev/sdb设备的UUID:

[root@ora10grac-dg rules.d]# scsi_id -g -v -s /block/sdb
36000c29e0aa117a4e812fb78d5fb3517
[root@ora10grac-dg rules.d]#                  

             然后,在/etc/udev/rules.d/60-raw.rules文件里添加配置规则如下:

[root@ora10grac-dg rules.d]# pwd
/etc/udev/rules.d
[root@ora10grac-dg rules.d]# ll
total 156
-rw-r--r-- 1 root root   515 Jul  3  2009 05-udev-early.rules
-rw-r--r-- 1 root root   900 Mar 31  2010 40-multipath.rules
-rw-r--r-- 1 root root 15647 Jul  3  2009 50-udev.rules
-rw-r--r-- 1 root root   471 Jul  3  2009 51-hotplug.rules
-rw-r--r-- 1 root root   143 Jul  4  2009 60-net.rules
-rw-r--r-- 1 root root  1088 Jun  6  2007 60-pcmcia.rules
-rw-r--r-- 1 root root   475 Feb 15 15:30 60-raw.rules
-rw-r--r-- 1 root root  9014 Apr  1  2010 60-wacom.rules
-rw-r--r-- 1 root root   129 Apr  1  2010 61-uinput-stddev.rules
-rw-r--r-- 1 root root   214 Apr  1  2010 61-uinput-wacom.rules
-rw-r--r-- 1 root root    61 Feb 15 15:32 65-raw-permissions.rules
-rw-r--r-- 1 root root  1823 Nov  5  2008 85-pcscd_ccid.rules
-rw-r--r-- 1 root root   114 Nov  5  2008 90-alsa.rules
-rw-r--r-- 1 root root    61 Jul  3  2009 90-dm.rules
-rw-r--r-- 1 root root    82 Apr  1  2010 90-hal.rules
-rw-r--r-- 1 root root   107 Jul  3  2009 95-pam-console.rules
-rw-r--r-- 1 root root   292 Apr  1  2010 98-kexec.rules
-rw-r--r-- 1 root root  2319 Jul 15  2008 bluetooth.rules
[root@ora10grac-dg rules.d]# cat 60-raw.rules 
# Enter raw device bindings here.
#
# An example would be:
#   ACTION=="add", KERNEL=="sda", RUN+="/bin/raw /dev/raw/raw1 %N"
# to bind /dev/raw/raw1 to /dev/sda, or
#   ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="1", RUN+="/bin/raw /dev/raw/raw2 %M %m"
# to bind /dev/raw/raw2 to the device with major 8, minor 1.
ACTION=="add", KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="36000c29e0aa117a4e812fb78d5fb3517",RUN+="/bin/raw /dev/raw/raw1 %N" 
[root@ora10grac-dg rules.d]# 

             其次, 在/etc/udev/rules.d/路径下添加一个权限控制文件,这里命名为65-raw-permissions.rules,其作用是确保操作系统重启后,oracle用户仍然有对裸设备操作的权限。否则,重启系统后,ASM 实例将不可用。

[root@ora10grac-dg rules.d]# cat 65-raw-permissions.rules 
KERNEL=="raw1", OWNER="oracle", GROUP="oinstall", MODE="660"
[root@ora10grac-dg rules.d]# 

             最后,重新加载规则,并启动udev:      

[root@ora10grac-dg rules.d]# /sbin/udevcontrol reload_rules
[root@ora10grac-dg rules.d]# /sbin/start_udev 
Starting udev: [  OK  ]
[root@ora10grac-dg rules.d]# ll /dev/raw/raw1 
crw-rw---- 1 oracle oinstall 162, 1 Feb 15 15:34 /dev/raw/raw1
[root@ora10grac-dg rules.d]# 

             至此,raw设备配置完毕!

     ③     创建ASM实例:

            首先,确认asm相应的软件包安装成功:

[root@ora10grac-dg ~]# rpm -qa | grep asm
oracleasm-support-2.1.3-1.el5
oracleasm-2.6.18-194.el5-2.0.5-1.el5
[root@ora10grac-dg ~]#

           从上,可以看到系统并没有安装asmlib包,因为我们是用裸设备作asm磁盘

           然后,配置ASM服务:

[root@ora10grac-dg ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@ora10grac-dg ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [  OK  ]
Scanning the system for Oracle ASMLib disks: [  OK  ]
[root@ora10grac-dg ~]# /etc/init.d/oracleasm listdisks
[root@ora10grac-dg ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [  OK  ]
[root@ora10grac-dg ~]# /etc/init.d/oracleasm listdisks
[root@ora10grac-dg ~]# /etc/init.d/oracleasm querydisk /dev/sd*
Device "/dev/sda" is not marked as an ASM disk
Device "/dev/sda1" is not marked as an ASM disk
Device "/dev/sda2" is not marked as an ASM disk
Device "/dev/sdb" is not marked as an ASM disk
Device "/dev/sdb1" is not marked as an ASM disk
Device "/dev/sdc" is not marked as an ASM disk
Device "/dev/sdc1" is not marked as an ASM disk
[root@ora10grac-dg ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@ora10grac-dg ~]#  

          从上面的输出中,可以看到asm服务会随着操作系统启动而加载,并且在没有用asmlib包配置asm磁盘的情况下,执行/etc/init.d/oracleasm listdisks命令是看不到磁盘的。

          其次,oracle用户登录图形界面,执行dbca,进入图形界面进行安装:

         

          单击Next,进入下一步:

            选择最后一项,配置ASM,单击Next:

     

                提示需要以root用户执行"/u01/app/oracle/bin/localconfig add"命令来启动CSS服务:

[root@ora10grac-dg ~]# /u01/app/oracle/bin/localconfig add
/etc/oracle does not exist. Creating it now.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized

Adding to inittab 
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        ora10grac-dg
CSS is active on all nodes.
Oracle CSS service is installed and running under init(1M)
[root@ora10grac-dg ~]# ps -ef | grep css
root     25911     1  0 16:06 ?        00:00:00 /bin/sh /etc/init.d/init.cssd run
oracle   26151 25911  4 16:06 ?        00:00:01 /u01/app/oracle/bin/ocssd.bin
root     26171  3352  0 16:07 pts/1    00:00:00 grep css
[root@ora10grac-dg ~]# 

                执行完上述命令之后,回到图形界面,单击OK,进入下一步:

               输入ASM实例的SYS用户的口令,单击Next:

                   提示,DBCA将创建ASM实例,单击OK,下一步:

                  这时,可以在操作系统上看到ASM实例的后台进程:

[root@ora10grac-dg ~]# ps -ef | grep asm_
oracle   26211     1  0 16:12 ?        00:00:00 asm_pmon_+ASM
oracle   26213     1  0 16:12 ?        00:00:00 asm_psp0_+ASM
oracle   26215     1  0 16:12 ?        00:00:00 asm_mman_+ASM
oracle   26217     1  0 16:12 ?        00:00:00 asm_dbw0_+ASM
oracle   26219     1  0 16:12 ?        00:00:00 asm_lgwr_+ASM
oracle   26221     1  0 16:12 ?        00:00:00 asm_ckpt_+ASM
oracle   26223     1  0 16:12 ?        00:00:00 asm_smon_+ASM
oracle   26225     1  0 16:12 ?        00:00:00 asm_rbal_+ASM
oracle   26227     1  0 16:12 ?        00:00:00 asm_gmon_+ASM
root     26233  3352  0 16:12 pts/1    00:00:00 grep asm_
[root@ora10grac-dg ~]# 

                  单击Create New,创建磁盘组,输入磁盘组的名称,oradata,冗余级别选择外部,选中磁盘/dev/raw/raw1,单击Ok:

                最后,单击Finish完成ASM实例的创建。

                我们,可以在SQL*PLUS的字符命令行下访问asm实例:

[root@ora10grac-dg ~]# su - oracle
[oracle@ora10grac-dg ~]$ export ORACLE_SID=+ASM
[oracle@ora10grac-dg ~]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.5.0 - Production on Wed Feb 15 16:20:45 2012

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

SQL> conn / as sysdba;
Connected.
SQL> show parameter spfile;

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
spfile                               string                            /u01/app/oracle/dbs/spfile+ASM
                                                                       .ora
SQL> show parameter asm

NAME                                 TYPE                              VALUE
------------------------------------ --------------------------------- ------------------------------
asm_diskgroups                       string                            ORADATA
asm_diskstring                       string
asm_power_limit                      integer                           1
SQL> col name for a30
SQL> select name,state,total_mb,free_mb from v$asm_diskgroup;

NAME                           STATE                               TOTAL_MB    FREE_MB
------------------------------ --------------------------------- ---------- ----------
ORADATA                        MOUNTED                                10236      10186

SQL> 

至此,我们的物理备库已经准备就绪,但是我们的物理备库还没有搭建出来,未完待续!下一篇,介绍物理备库搭建的详细步骤,敬请期待!!!

顶一下
(0)
100%
踩一下
(0)
100%

2 篇回应 (访客:2 篇, 博主:0 篇)

  1. sam wang 2012-12-03

    根据客户需求,我们的环境需要在一个RAC节点(比如:oracle-rac1)上配置Datagurad。

    我的问题是:

    如果物理备库和RAC节点1(oracle-rac1)是同一台机器,并且备库采用local disk方式存储,哪些步骤需要变化?如何变化?

    可否就我的问题按照你的Oracle 10g RAC 配置物理dataguard系列1-4分别指出有哪些不同,能给出详细的步骤就再好不过了!

    首先在此谢过!

    #-49楼
  2. Gao 2012-24-10

    做得不错,很详细,希望看看你后续章节!!!

    #-48楼

插入图片

NOTICE1:请申请gravatar头像,没有头像的评论可能不会被回复|头像相关帮助!