Bash Shell Scripting: Comparing Directories

Hey guys,I've been tasked to create a bash shell script to compare directories and only list out the differences between the two directory.Does anyone have any ideas or pointers of how to go about this problem?Many thanks,xboxrulz

Find out the difference between the file listing in two directories? Or it should even list out the difference between each files in the two directories?

 

If it is the difference between the file listing, it can be done easily than the other requirement. Here is what I can think top of my mind:

 

1. Ensure you are not in the directories that needs to be compared

 

cd [path-to-some-temporary-location]

 

2. Get the single line file listing of both directories into temporary files

 

dir -1 [path/first-directory] > dirlist1

dir -1 [path./second-directory] > dirlist2

 

3. Do a compare between these two:

 

diff dirlist1 dirlist2 > difference.txt

 

If you want th differences to be printed on scree - ofcourse, remove the redirection to difference.txt

 

4. Clean up!

 

rm dirlist1 dirlist2

 

The difference.txt or the output (if you removed redirection) would list the files which are in the first directory and not in second, and vise versa. Put all these commands in a .bsh with a shebhang at top, change permissions to let appropriate people execute the script, and you are good to go.

 

Now, ofcourse, there might be possibility of stuffing all these into one command using pipes and tees (I am lazy to figure out exactly how it can be done).

Edited October 26, 2008 by Vyoma (see edit history)

First point.For honesty purposes, just tell me something. Is this something you want to do for a friend, or is this a homework your teacher asked you to perform ? Because if it's a homework, your teacher "at order 0" wants you to think by yourself, an you have to do the job quite alone.Of course, if your teacher is smart, "at order 1" he wants to see if you are able to do some googling.And "at order 2" he wants to know if you have some help on forums."At order 3" the guys on the forum can do mistakes and you must fix my bugs.Second point. Here we are at in the "open" world. That means that if you ask two people the same question, you will have two different answers. And both of them well be righ.My first idea was exactly what vyoma says, two lists and a "diff" to see the difference. seeing that, my second thought is "OK, I will do it differently.Here is my script, I wold like you to test it.# Let us suppose that we want to see if the files in folder1 are also in folder2.cd folder1for i in `ls`doif [ if ../folder2/$i]then #do nothing because I want to know the non-existing files present=1else echo "folder2/"$i file is missing ls -l $i >> /tmp/size_of_the_impacted_filesfi# end of job# This shell_script is copywrited, only Yordan friends may use itAnd remember, using my shell-scripts costs 15% of the money you earn with my scripts.RegardsYordan

Based on what xboxrules actually wants, that is, if his answer to my second question was yes:

 

Or it should even list out the difference between each files in the two directories?

Then I think yordan's solution is more elegant. All it needs next, are two things:

1. List of files in folder2 and not in folder1 - implementation will be exactly same as what yordan wrote

2. Find out the difference bettween files which are present in both folders - To implement this, in one of the loops, when you deduce that the files are present in both directories, run a 'diff folder1/$i folder2/$i'

Find out the difference between the file listing in two directories? Or it should even list out the difference between each files in the two directories?

 

If it is the difference between the file listing, it can be done easily than the other requirement. Here is what I can think top of my mind:

 

1. Ensure you are not in the directories that needs to be compared

 

cd [path-to-some-temporary-location]

 

2. Get the single line file listing of both directories into temporary files

 

dir -1 [path/first-directory] > dirlist1

dir -1 [path./second-directory] > dirlist2

 

3. Do a compare between these two:

 

diff dirlist1 dirlist2 > difference.txt

 

If you want th differences to be printed on scree - ofcourse, remove the redirection to difference.txt

 

4. Clean up!

 

rm dirlist1 dirlist2

 

The difference.txt or the output (if you removed redirection) would list the files which are in the first directory and not in second, and vise versa. Put all these commands in a .bsh with a shebhang at top, change permissions to let appropriate people execute the script, and you are good to go.

 

Now, ofcourse, there might be possibility of stuffing all these into one command using pipes and tees (I am lazy to figure out exactly how it can be done).

Does the $* variable allow the the user to enter "/home/user/path" as one variable or will it split it to home, user and path separately per $. Like home being $1, user being $2 and path being $3?

 

First point.

For honesty purposes, just tell me something. Is this something you want to do for a friend, or is this a homework your teacher asked you to perform ?

Because if it's a homework, your teacher "at order 0" wants you to think by yourself, an you have to do the job quite alone.

Of course, if your teacher is smart, "at order 1" he wants to see if you are able to do some googling.

And "at order 2" he wants to know if you have some help on forums.

"At order 3" the guys on the forum can do mistakes and you must fix my bugs.

 

Second point. Here we are at in the "open" world. That means that if you ask two people the same question, you will have two different answers. And both of them well be righ.

My first idea was exactly what vyoma says, two lists and a "diff" to see the difference. seeing that, my second thought is "OK, I will do it differently.

Here is my script, I wold like you to test it.

# Let us suppose that we want to see if the files in folder1 are also in folder2.

cd folder1

for i in `ls`

do

if [ if ../folder2/$i]

then

#do nothing because I want to know the non-existing files

present=1

else

echo "folder2/"$i file is missing

ls -l $i >> /tmp/size_of_the_impacted_files

fi

# end of job

# This shell_script is copywrited, only Yordan friends may use it

 

And remember, using my shell-scripts costs 15% of the money you earn with my scripts.

Regards

Yordan

It is for homework, thing is I didn't quite get how to approach this. Obviously I'm not going to copy code, that's just plain wrong. I just want ideas and adapt them.

 

lol, and i don't think anyone is gonna really make any money out of using a directory comparison script, but i never know lol.

 

Based on what xboxrules actually wants, that is, if his answer to my second question was yes:

Then I think yordan's solution is more elegant. All it needs next, are two things:

1. List of files in folder2 and not in folder1 - implementation will be exactly same as what yordan wrote

2. Find out the difference bettween files which are present in both folders - To implement this, in one of the loops, when you deduce that the files are present in both directories, run a 'diff folder1/$i folder2/$i'

It only lists out the files that are for example in folder 1 but not 2 and vice versa but the list view must be like what ls -l would list so dircmp won't cut it unfortunately...

 

Like this:

-rw-r--r-- 1 xboxrulz staff 16 Oct 27 23:57 differences.txt

-rw-r--r-- 1 xboxrulz staff 19 Oct 27 23:57 dirlist1

-rw-r--r-- 1 xboxrulz staff 9 Oct 27 23:57 dirlist2

drwxr-xr-x 2 xboxrulz staff 152 Oct 27 21:16 folder1

drwxr-xr-x 2 xboxrulz staff 110 Oct 27 21:17 folder2

... but showing the different files.

 

Thanks for the pointers guys, that just gave me some clue to this confusing work.

 

xboxrulz

 

{EDIT} P.S: I think Vyoma's idea is easier to code, let me try that out first ...

Does the $* variable allow the the user to enter "/home/user/path" as one variable or will it split it to home, user and path separately per $. Like home being $1, user being $2 and path being $3?

...

I am not sure about $* - will need to look it up. But basically, $1, $2 etc refer to argument list passed to the shell script.

So, if you pass:

> script.sh "/home/user/path"

Then, when you say $1, it will be equal to "/home/user/path" - it will not be split up.

 

...

It only lists out the files that are for example in folder 1 but not 2 and vice versa but the list view must be like what ls -l would list so dircmp won't cut it unfortunately...

...

Yes, what yordan gave would only list files that are in folder 1 but not in folder 2, and implementing the other would give you the other set too - correct.

(Is there a dircmp command? Not sure there is something like that). So, if you want the listing to be something like what 'ls -l' would give, just use the result of yordan's solution to get it.

 

Instead of what the code (yordan's)

...elseecho "folder2/"$i file is missingls -l $i >> /tmp/size_of_the_impacted_filesfi# end of job...

write it as

...elsels -l $i | tail -1fi# end of job...

The "ls -l $i | tail -1" will give each of the missing file in the "ls -l" form. (It is a bit tricky here- you might want to prefix $i with the folder path it is present in).

 

...

{EDIT} P.S: I think Vyoma's idea is easier to code, let me try that out first ...

Just try out a code and see if it works - if it does not, post it and we can help.

By the way, yes, $1 works.However, your script should first check that $1 is not empty and $2 is not empty.And the, in my script you replace the word "folder1" by $1 and the word "folder2" by $2$* should work, it is supposed to pass $1, $2, $2 and $4. But then your script must manage that, and say something like "one=$1; shift; two=$2; three=$3; if [ -Z $3 ] " etc...

I'm not sure whether I'm one of yordan's friends, but I am definitely someone who needs more exposure to scripts, so I tried using it. The result was bash: [: missing `]'

 

I visited Advanced Bash-Scripting Guide and the man page for bash, to remind me how to construct an if-test. It reminded me of enough that I thought I could fix it. I tested it, and bash no longer complains. [Gee, the forum just cleaned up my alternative term for "complains": b**ches. I had no idea it did that! ]

 

In one short line, a running script is

 

for i in `ls'; do if [ -e ../u/$i ]; then echo "first"; else echo "second'; fi done

 

The things that needed to be fixed were (1) second if -> -e (or -a), (2) blank before the closing bracket (]) and (3) "done" at the end of the script.

 

Thanks for the exercise -- it's been a long time since I've used all those script elements!

Ok, I had to scrap it since my professor highly discouraged using diff because it's too specific and too sensitive to differences.

He told me to use
for file in $(ls -a $1)
if (! -d $1/$file)
to check the differences of the directories

So I ended up with scripting this.

#This code is GPL v3for file in $(ls -a $1)   do		if [ ! -e $2/$file ]			then				ls -ld $1/$file		fi   done   for file in $(ls -a $2)    do		if [ ! -e $1/$file ]			then				ls -ld $2/$file		fi   done

and this script works flawlessly.

Thanks for the pointers guys, really helped,
xboxrulz

for i in `ls'; do if [ -e ../u/$i ]; then echo "first"; else echo "second'; fi done

Just kidding, but two remarks.Putting everything in a single line seems nicer, but it's far less easy to read it, and find an error if there happen to be one.
Instead, writing
if [ something ]
then
do the first step
else
do the second thing
fi

The second way is more readable.
And, of course, it's easier to see the error.
I see for i in `ls'; do
it should be for i in `ls`
and of course bash, exactly as ksh , sh and cs, hates reading standard quotes (') instead of back quotes (`)

Just kidding, but two remarks.

Putting everything in a single line seems nicer, but it's far less easy to read it, and find an error if there happen to be one.

Agreed. But I'm lazy, and I was typing your script directly into a console. When bash complained, <up-arrow> got me that one-line version complete with semicolons, which I could easily edit to identify and correct the bugs.

 

Then, when I had running code, I was lazy again, and just copied the running code and pasted it into my post. That way, I could be sure I wouldn't break anything by "pretty-printing" the code.

 

Anyway, this thread is doing what it is supposed to -- teaching each of us little bits of useful information.

 

I am now scratching my head over xboxrulz's final version, with for file in $(ls -a $1). I am trying to figure out how that is interpreted, and will have to try it.

 

Very educational thread. Thanks guys!

Docduke,

 

for file in $(ls -a $1) basically tells the script to grab all (-a) the file/folder entries including all hidden folders in the folder $1 (which is a variable argument).

 

xboxrulz

Maybe this is a new function, but now you can compare directories simply whitdiff /direone/ /diretwo/it compares the files in each on and the contents.

Maybe this is a new function, but now you can compare directories simply whit
diff /direone/ /diretwo/

it compares the files in each on and the contents.

Quite true.
Unfortunately, the teacher asked the student not to use "diff", in order to learn how to manage recursive commands and how to pass variables to scripts.