Merge Data in Linux - How to do it ?

There are situations when you have scattered data and you want to combine it in one spot called merging. It's possible that you divided a single file into numerous files and now wish to merge them back together or that you have several log files that you'd like to merge into one. In Linux, merging numerous text files into a single file is simple. 

The process of merging two or more data sets into a single data set is known as data merging. When you have raw data stored in numerous files, workbooks, or data tables that you want to analyze all at once, this approach is usually required.

Here at LinuxAPT, we shall look into different ways to merge data in one place through different commands on Ubuntu 20.04 Linux distribution.

Also, you will see how to use "cat", "sed" and "merge" commands to merge data on Ubuntu (Linux OS).

Different methods of Merging data in Linux:

Some procedures are mentioned below to merge data using some commands on Ubuntu:

• Merge data using cat command.
• Merge data using sed command.
• Merge data using merge command.
• Merge data using "For loop".

1. Merge Data Using Cat Command

The cat stands for concatenate; it is preinstalled in new Ubuntu versions but if you are using an older version you need to install it. It is a commonly used command that reads all data from a file and outputs its content on the terminal screen. It allows us to generate, view, and combine files. When you use the cat command to display the contents of huge text files to the terminal, it will muck up your terminal and make navigation difficult.

The below-mentioned command will merge the data of "linux1" and "linux2" and display it on screen in the same order filenames are placed:

$ cat linux1.txt linux2.txt

2. Merge Data and Store in a File

Now we can also merge data from multiple files and store it in another file using the cat command and redirection operator ">". The below-mentioned command will merge data of "linux1" and "linux2" and store it in "merged_linux" using the cat command:

$ cat linux1.txt linux2.txt > merged_linux.txt

If the file does not exist, the cat command will create it first. Instead of adding at the end, the single redirection operator will overwrite the file, you have to use a double redirection operator if you want to append new text at the end of the file without overwriting.

3. Merge Data Using Sed Command

In Linux, the SED pre-installed command is abbreviated as stream editor but if it is not installed you can install it and it can perform a variety of file operations such as searching, finding, and replacing, insertion, and deletion. The SED command is a popular Linux command used for replacement or for finding and replacing. You can modify files without opening them using SED, which is a far faster way to find and replace anything in a file than opening it in VI editor first and then altering it.

The sed command, which is usually used for text manipulation and transformation, can be used for merging files/data. ">". Below mentioned command will merge data of "linux1" and "linux2" and store it in "merged_linux" using sed command and redirection operator ">":

$ sed h linux1.txt linux2.txt > merged_linux1.txt

4. Merge data using merge command

Merge command also merges the data of two files and stores them in a new file but it works differently than any other merging command. Merge compares three files, an original and two changed versions of the original, line by line, seeking to reconcile conflicts between the two sets of modifications to create a single, combined file that represents the changes of both files. The "merge" command is not preinstalled you need to install it by the below-mentioned command:

$ sudo apt install rcs

The below-mentioned command will merge linux1.txt and linux2.txt into "merged_linux.txt" using the merge command:

$ merge merge_linux2.txt linux1.txt linux2.txt 

The linux1.txt and linux2.txt are two files that merge different parts in "merge_linux2.txt", you need to create "merge_linux2.txt" first.

There is a conflict between "<<<<<<<" and ">>>>>>>".

5. Merge data using "For loop"

The "for loop" can eliminate the need to explicitly state the file names. This will only function if the filenames are consistent. In our situation, the file names are formatted as follows: linux{1,2}.txt

Below mentioned command will merge data of "linux1" and "linux2" and store it in "merged_linux" using for loop and redirection operator ">":

$ for i in {1,2}; do cat “linux$i.txt” >> merged_linux3.txt; done

[Need help in fixing Linux system issues ? We can help you. ]