Skip to main content

how to add storage to Hadoop DataNode

Hi All,
This is a very simple operation intend to be very confusing as I saw in other posts I read.
What you will need to add storage to hadoop is a new disk, mounted on a new volume (directory)

After you got a new disk, ask you system administrator to mount him on a new desired directory:
[root@RHEL7datanode1 ~]# mkdir -p /hadoop/add_disk/
[root@RHEL7datanode1 ~]# chown -R hdfs:hadoop /hadoop

[root@RHEL7datanode1 ~]# su - hdfs

now you need to change conf file and add this new folder to the hdfs-site.xml

vi /usr/hdp/2.3.0.0-2557/hadoop/conf/hdfs-site.xml

    <property>
      <name>dfs.datanode.data.dir</name>
      <value>/data2/hdp,/data1/hdp,/hadoop/add_disk</value>
    </property>


Restart datanode:

[hdfs@RHEL7datanode1 ~]$ /usr/hdp/2.3.0.0-2557/hadoop/sbin/hadoop-daemon.sh stop datanode
stopping datanode

[hdfs@RHEL7datanode1 ~]$ /usr/hdp/2.3.0.0-2557/hadoop/sbin/hadoop-daemon.sh startp datanode
starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-RHEL7datanode1.out


After the datanode is up look for something like this in your log file: (to verify it's OK)

[root@RHEL7datanode1 hdfs]# tail -f /var/log/hadoop/hdfs/hadoop-hdfs-namenode-RHEL7datanode1.log

2016-11-10 12:56:59,673 INFO  blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateStorage(856)) - Adding new storage ID DS-cf75-40f2--f00574d388b0 for DN 10.212.125.124:50010


This command will show you the new capacity size:
[hdfs@RHEL7datanode1 ~]$ hdfs dfsadmin -report

Comments

Post a Comment

Popular posts from this blog

ORA-27104: system-defined limits for shared memory was misconfigured

I faced this error while trying to restore & recover of a PDB (pluggable database) part of the log file and the solution is described here below: log: initialization parameters used for automatic instance: db_name=CDB db_unique_name=gbux_pitr_PDB1_CDB compatible=12.1.0.2.0 db_block_size=8192 db_files=200 diagnostic_dest=/oracle/app/oracle _system_trig_enabled=FALSE sga_target=1888M processes=200 db_create_file_dest=/oracle/auxilary log_archive_dest_1='location=/oracle/auxilary' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 07/31/2016 16:22:20 RMAN-04014: startup failed: ORA-27104: system-defined limits for shared...
6 comments
Read more

mount.nfs: backgrounding

if you face this kind of error with your remote NFS: [root@Vertica000 ~]# mount /files/application/Rremote3 mount.nfs: backgrounding "10.0.0.2:/files/application/remoteFiles" mount.nfs: mount options: "bg,hard,nointr,rsize=65536,wsize=65536,tcp,actimeo=0,vers=3,timeo=600,addr=10.0.0.2" look for the problem in the log file: cat /var/log/messages | grep mount  mount to NFS server '10.0.0.2' failed: timed out, retrying Solution: In most of the cases, you have a problem with your iptables in the destination server login as root to dest server (10.0.0.2) in my case and type this command: iptables --flush  the go back to your origin server to try remount the problematic NFS file system of course this is in case nfs server was installed and functioning properly. Good luck.
1 comment
Read more

ora-65035 while create pluggable database from another one

or : ORA-21000: error number argument to raise_application_error of -65035 is out of range Cause: this error caused because of an unrecoverable transaction in the source DB. Solution: 1. either stop and start your pluggable source DB 2. identify the open transaction and kill it if you can. use this query to do so: select * from v$transaction t,v$session  s where t.ses_addr = s.saddr;
Post a Comment
Read more