Stephen Royle

Here is a quick method for checking parity between two directories.

Let's say we have two directories `dir1` and `dir2`. They are large and
have thousands of files and subdirectories. How can we check that they
have the same contents? I found myself in this situation recently during
a server migration.

## Method 1: rsync {#method-1-rsync .wp-block-heading}

To move the files in the first place we can do
`rsync -av --checksum path/to/dir1/ path/to/dir2` which will copy the
contents of `a` into the `b` directory. Use of the checksum flag means
that the hashes are checked during transfer. If `b` is a new directory,
or you use the --delete flag to erase anything else in `b` then this
means the transfer is done and that it is checked. Barring any errors
both directories will be the same. If you want to just see what would
have happened with the command, use --dry-run.

The `rsync` command can be run again to check that everything went OK
and that there are no more files to update or change. The problem is
that with a large directory that has many files and subdirectories, this
is slow, especially over a network connection.

## Method 2: diff {#method-2-diff .wp-block-heading}

Using diff to check for differences between the contents of `dir1` and
`dir2` is a great way to check parity.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .bash; .title: .; .notranslate title=""}
diff -rq path/to/dir1 path/to/dir2
```
:::

This will list any files that are present/absent in one directory or the
other and it will also show which files have changed.

Again the problem here is speed. If the diff is done in the shell of a
computer with two network volumes, particularly over a slow connection,
this can take a long time.

## Method 3: python {#method-3-python .wp-block-heading}

Python has a library filecmp which could help us.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .python; .title: .; .notranslate title=""}
import filecmp

dir1 = 'path/to/dir1'
dir2 = 'path/to/dir2'
filecmp.dircmp(dir1,dir2).report_full_closure()
```
:::

We can get lists of contents and it works fast, but the results are
difficult to make sense of.

## Method 4: easy/quick method {#method-4-easyquick-method .wp-block-heading}

Let's say we are pretty sure that the directories are the same, but we
just want to make sure. Imagine you did the migration a few days ago but
weren't sure if anything has changed in `dir1` or `dir2` in that time.
If others have read-write access to `dir1` or `dir2` there might be
files added or removed that you don't know about.

Here's a simple solution in zsh.

Get a list of files in both directories and put them in two text files
on the desktop.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .bash; .title: .; .notranslate title=""}
cd path/to/dir1
find -L . > ~/Desktop/dir1.txt
cd path/to/dir2
find -L . > ~/Desktop/dir2.txt
```
:::

Change directory means that the relative paths of all files and folder
listed in the the text file are comparable. The two outputs we get for a
dummy set of files (created using this
[script](https://bip.weizmann.ac.il/course/python/PyMOTW/PyMOTW/docs/filecmp/index.html))
looks like this

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
.
./not_the_same
./dir_only_in_dir1
./file_only_in_dir1
./common_file
./file_in_dir1
./common_dir
./common_dir/dir2
./common_dir/dir2/not_the_same
./common_dir/dir2/common_file
./common_dir/dir2/file_in_dir1
./common_dir/dir2/dir_only_in_dir2
./common_dir/dir2/common_dir
./common_dir/dir2/file_only_in_dir2
./common_dir/dir1
./common_dir/dir1/not_the_same
./common_dir/dir1/dir_only_in_dir1
./common_dir/dir1/file_only_in_dir1
./common_dir/dir1/common_file
./common_dir/dir1/file_in_dir1
./common_dir/dir1/common_dir
```
:::

and

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
.
./not_the_same
./common_file
./file_in_dir1
./dir_only_in_dir2
./common_dir
./common_dir/dir2
./common_dir/dir2/not_the_same
./common_dir/dir2/common_file
./common_dir/dir2/file_in_dir1
./common_dir/dir2/dir_only_in_dir2
./common_dir/dir2/common_dir
./common_dir/dir2/file_only_in_dir2
./common_dir/dir1
./common_dir/dir1/not_the_same
./common_dir/dir1/dir_only_in_dir1
./common_dir/dir1/file_only_in_dir1
./common_dir/dir1/common_file
./common_dir/dir1/file_in_dir1
./common_dir/dir1/common_dir
./file_only_in_dir2
```
:::

This step is seriously fast. On the large directories I was scanning, it
took just a few minutes for `find` versus hours for `rsync` or `diff`.

Now we can sort the text files into alphabetical order and put the
contents each into a new file:

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .bash; .title: .; .notranslate title=""}
cd ~/Desktop
sort dir1.txt > dir1sort.txt
sort dir2.txt > dir2sort.txt
```
:::

Then we can just `diff` the text files by doing
`diff dir1sort.txt dir2sort.txt`

However, I really like [diff2html](https://diff2html.xyz) for a more
visual way to observe the output. Using:

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .bash; .title: .; .notranslate title=""}
diff -u dir1sort.txt dir2sort.txt | diff2html -i stdin
```
:::

We get this:

<figure class="wp-block-image size-large">
<img
src="https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?resize=640%2C299&amp;ssl=1"
class="wp-image-2596" loading="lazy" decoding="async"
data-attachment-id="2596"
data-permalink="https://quantixed.org/2021/12/18/tips-from-the-blog-xvi-fast-check-of-two-directories/screenshot-2021-12-18-at-20-47-13/"
data-orig-file="https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?fit=1478%2C692&amp;ssl=1"
data-orig-size="1478,692" data-comments-opened="1"
data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}"
data-image-title="Screenshot-2021-12-18-at-20.47.13"
data-image-description="" data-image-caption=""
data-medium-file="https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?fit=300%2C140&amp;ssl=1"
data-large-file="https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?fit=640%2C299&amp;ssl=1"
srcset="https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?resize=1024%2C479&amp;ssl=1 1024w, https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?resize=300%2C140&amp;ssl=1 300w, https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?resize=768%2C360&amp;ssl=1 768w, https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?w=1478&amp;ssl=1 1478w, https://i0.wp.com/quantixed.org/wp-content/uploads/2021/12/Screenshot-2021-12-18-at-20.47.13.png?w=1280&amp;ssl=1 1280w"
sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" width="640"
height="299" />
</figure>

The sorting step helps with diff, because contiguous blocks of flies in
subdirectories will be grouped together following the sort.

The example I'm showing here is very simple and the power of this method
comes when a complicated directory tree needs to be quickly compared.

---

This post is part of an occasional series of tech tips.


Here is a quick method for checking parity between two directories. Let’s say we have two directories dir1 and dir2. They are large and have thousands of files and subdirectories. How can we check that they have the same contents? I found myself in this situation recently during a server migration.

Messaggi di Rogue Scholar

Tips From The Blog XVI: fast check of two directories