PROVIDING DYNAMIC STORAGE TO HADOOP
Hello! folks here comes a new blog which shows how to implement the dynamic storage in the Hadoop cluster which is actually important in today’s world. As the huge amount of data is increasing in micro seconds so the big data has emerged as a technology in today’s world with its tool popularly known as Hadoop.
Now all knows about the server of the company and at the back the MNCs need a huge storage to store the data of the clients they have so they cannot let their servers down and increase their storage and then again start their servers they will face a huge loss because of this activity and after that the data to be uploaded again to the storage which will consume a lot of time.
So, to get rid of all these problems in the company there is a concept of Logical Volume Management which can be implemented in the companies for the server storage management and as there is a huge amount of data which is generated in seconds so it is required to integrate this concept with the Big Data technology.
Before commencing this setup if you are new to this technology you should first understand the concept of LVM and the process with it works.
As this is the hadoop setup one should know how to create hadoop and if you are new to the technology you should refer to this blog shared below.
Now let us move to build the setup. We just need to see the steps given below.
Disk Status
First we need to check the status of the disk and that can be checked by the command shown below.
Now here we have attached two hard disks in the Operating System which will be used as an external storage to create Physical Volume.
For implementing the whole setup we need to first install the software LVM2 which can be installed by a command shown below.
Creating Physical Volume
Now we will take both the hard disks and will create the physical disks with both of them which can be done with the command shown below.
1st Volume
2nd Volume
We can view both the physical disks which can be done by the command shown below.
Creating Volume Group
Now when the physical volumes are created we need to make a group by combining both the physical hard disks or we can say we can make a volume group which can be done by a command shown below.
You can view the volume group created by the command shown below.
Creating Logical Volume
Now we need to create a logical volume or it can also be called as a partition which will be used as a storage to store the data of the clients.
You can view it by the command shown below.
Format the Partition or Logical Volume ( LV )
Now once the partition is created before using it we need to introduce the filesystem to the partition or disk created so that it can become cappable to have the index or node table.
It can be done by a command shown below.
Mounting of Disk
It is one of the most important step to get rid of data loss. Once we mount the folder the data will be seem to be going in the folder but the data will directly go to the logical volume seamlessly and data can be used and will remain as the backup in the disk.
This can be done by two steps shown below.
1.Make the directory a shown below :
2.Mount the folder with the disk as shown below :
After doing all of the above process let us now check the status of the disk by a command shown below.
Now one has to just edit the folder in hdfs-site.xml file and write the name of folder which is mounted with the folder as shown below.
Once done with it save the file and just restart the datanode by the command written below :
=> hadoop-daemon.sh start datanode
Check the status by the command :
=> jps
Extending Logical Volume
Now as per the requirement if we want to increase or extend the storage we can do it in a way shown below :
Once it is done you need to format the extended partition and that can be done by the command shown below :
Now once it is done just go to the dashboard as shown below:
Just refresh the page and you will see that 5 GB is extended in the storage as shown below.
Extending the Volume Group
First of all we need to attach a new hard disk to the Operating system and ir=t can be checked by the command shown below.
Now we need to create the Physical Volume as shown below.
Once it is completed then increasing the Volume Group can be done with so ease by just running the command shown below.
You can view it by the command shown below.
