Skip to main content

how to add storage to Hadoop DataNode

Hi All,
This is a very simple operation intend to be very confusing as I saw in other posts I read.
What you will need to add storage to hadoop is a new disk, mounted on a new volume (directory)

After you got a new disk, ask you system administrator to mount him on a new desired directory:
[root@RHEL7datanode1 ~]# mkdir -p /hadoop/add_disk/
[root@RHEL7datanode1 ~]# chown -R hdfs:hadoop /hadoop

[root@RHEL7datanode1 ~]# su - hdfs

now you need to change conf file and add this new folder to the hdfs-site.xml

vi /usr/hdp/2.3.0.0-2557/hadoop/conf/hdfs-site.xml

    <property>
      <name>dfs.datanode.data.dir</name>
      <value>/data2/hdp,/data1/hdp,/hadoop/add_disk</value>
    </property>


Restart datanode:

[hdfs@RHEL7datanode1 ~]$ /usr/hdp/2.3.0.0-2557/hadoop/sbin/hadoop-daemon.sh stop datanode
stopping datanode

[hdfs@RHEL7datanode1 ~]$ /usr/hdp/2.3.0.0-2557/hadoop/sbin/hadoop-daemon.sh startp datanode
starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-RHEL7datanode1.out


After the datanode is up look for something like this in your log file: (to verify it's OK)

[root@RHEL7datanode1 hdfs]# tail -f /var/log/hadoop/hdfs/hadoop-hdfs-namenode-RHEL7datanode1.log

2016-11-10 12:56:59,673 INFO  blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateStorage(856)) - Adding new storage ID DS-cf75-40f2--f00574d388b0 for DN 10.212.125.124:50010


This command will show you the new capacity size:
[hdfs@RHEL7datanode1 ~]$ hdfs dfsadmin -report

Comments

Post a Comment

Popular posts from this blog

Missing command :fuser Prerequisite check "CheckSystemCommandAvailable" failed

When trying to apply patch you may face this error the solution is described below. ERROR: bash-4.2$ ../opatch apply Oracle Interim Patch Installer version 12.2.0.1.5 Copyright (c) 2016, Oracle Corporation.  All rights reserved. Oracle Home       : /oracle/app/oracle/product/12.1.0 Central Inventory : /oracle/app/oracle/oraInventory    from           : /oracle/app/oracle/product/12.1.0/oraInst.loc OPatch version    : 12.2.0.1.5 OUI version       : 12.1.0.2.0 Log file location : /oracle/app/oracle/product/12.1.0/cfgtoollogs/opatch/opatch2016-07-27_16-27-44PM_1.log Verifying environment and performing prerequisite checks... Prerequisite check "CheckSystemCommandAvailable" failed. The details are: Missing command :fuser Prerequisite check "CheckSystemCommandAvailable" failed. The details are: Missing command :fuser Prerequisite ...
1 comment
Read more

ORA-27104: system-defined limits for shared memory was misconfigured

I faced this error while trying to restore & recover of a PDB (pluggable database) part of the log file and the solution is described here below: log: initialization parameters used for automatic instance: db_name=CDB db_unique_name=gbux_pitr_PDB1_CDB compatible=12.1.0.2.0 db_block_size=8192 db_files=200 diagnostic_dest=/oracle/app/oracle _system_trig_enabled=FALSE sga_target=1888M processes=200 db_create_file_dest=/oracle/auxilary log_archive_dest_1='location=/oracle/auxilary' enable_pluggable_database=true _clone_one_pdb_recovery=true #No auxiliary parameter file used starting up automatic instance CDB RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 07/31/2016 16:22:20 RMAN-04014: startup failed: ORA-27104: system-defined limits for shared...
6 comments
Read more

ora-65035 while create pluggable database from another one

or : ORA-21000: error number argument to raise_application_error of -65035 is out of range Cause: this error caused because of an unrecoverable transaction in the source DB. Solution: 1. either stop and start your pluggable source DB 2. identify the open transaction and kill it if you can. use this query to do so: select * from v$transaction t,v$session  s where t.ses_addr = s.saddr;
Post a Comment
Read more