Stephen Royle

I'm not a big movie person. Nonetheless I have a media library with
quite a few films in and I wondered how many "films to see before you
die"-type movies I had in the collection, and how many were missing. **I
used R to find the answers.**

I've described previously how to get a [plain text dump of a Plex
database](https://quantixed.org/2022/09/11/multiplex-comparing-plex-libraries/){data-type="post"
data-id="2816"} using
[WebTools-NG](https://github.com/WebTools-NG/WebTools-NG). I did that
for the Movies library of my Plex Media Server. Now, for the list of
"films to see before you die". I searched a bit and found a few text
files which claimed to be meta-rated as the best. I was a bit suspicious
about these. In the end, I figured I should just use to the IMDB's Top
250 Movies, which could be scraped with `rvest`.

## The code {#the-code .wp-block-heading}

Let's get the Top 250 movies:

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
library(rvest)
library(XML)
library(xml2)
library(fuzzyjoin)
library(dplyr)

# IMDB Top 250 Movies are here
url <- "http://www.imdb.com/chart/top?ref_=nv_wl_img_3"
page <- read_html(url)

movie.nodes <- html_nodes(page,'.titleColumn a')
movie.name <- html_text(movie.nodes)
sec <- html_nodes(page,'.secondaryInfo')
# to get the year we need to remove ) and ( and then get text)
year <- as.numeric(gsub(")","",gsub("\\(","",html_text( sec ))))

rating.nodes <- html_nodes(page,'.imdbRating strong')
rating <- as.numeric(html_text(rating.nodes))

imdb <- data.frame(Title = movie.name,
                   Year = year,
                   Rating = rating)
```
:::

Now we have a data frame of the movies, with Title, Year and the IMDB
rating.

We can load in the Plex library so that we can match them up, but we
don't need all the data.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
libfile <- file.choose()
libdf <- read.delim(libfile,sep = "|")

# we only need title and year
pms <- libdf %>% 
  select(Title,Year)
```
:::

Now we have two data frames to perform the matching.

The first issue is that we can't simply use the titles for matching
because remakes and different versions of movies will cause a mismatch.
To get around this we can use Title and Year as a combination for
*fuzzy* matching.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
# to match movies, we need Title-Year combination
imdb$titleyear <- paste(imdb$Title, imdb$Year)
pms$titleyear <- paste(pms$Title,pms$Year)

# fuzzy matching
match <- stringdist_join(imdb, pms, 
                         by = 'titleyear',
                         mode ='left',
                         method = "jw",
                         max_dist = 99, # could set this a lot lower 
                         distance_col = 'dist') %>% 
  group_by(titleyear.x) %>% 
  slice_min(order_by = dist, n = 1) # gives the best match found
```
:::

Fuzzy matching is needed because a simple string comparison will get
derailed pretty easily by capitalisation and other minor issues. So we
need something a little more forgiving to do the matching.

Now we can have a look at the matches by typing `match`

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
> # have a look at matches
> match
# A tibble: 254 × 8
# Groups:   titleyear.x [250]
   Title.x               Year.x Rating titleyear.x                Title.y                   Year.y titleyear.y           dist
   <chr>                  <dbl>  <dbl> <chr>                      <chr>                     <chr>  <chr>                <dbl>
 1 12 Angry Men            1957    9   12 Angry Men 1957          12 Monkeys                1995   12 Monkeys 1995     0.282 
 2 12 Years a Slave        2013    8.1 12 Years a Slave 2013      Oz the Great and Powerful 2013   Oz the Great and P… 0.348 
 3 1917                    2019    8.2 1917 2019                  Cats                      2019   Cats 2019           0.296 
 4 2001: A Space Odyssey   1968    8.3 2001: A Space Odyssey 1968 2001: A Space Odyssey     1968   2001: A Space Odys… 0     
 5 3 Idiots                2009    8.3 3 Idiots 2009              The Incredibles           2004   The Incredibles 20… 0.286 
 6 A Beautiful Mind        2001    8.2 A Beautiful Mind 2001      Beautiful Noise           2014   Beautiful Noise 20… 0.312 
 7 A Clockwork Orange      1971    8.2 A Clockwork Orange 1971    A Clockwork Orange        1972   A Clockwork Orange… 0.0290
 8 A Separation            2011    8.2 A Separation 2011          Separado!                 2010   Separado! 2010      0.189 
 9 Aladdin                 1992    8   Aladdin 1992               Aladdin                   1992   Aladdin 1992        0     
10 Alien                   1979    8.4 Alien 1979                 Alien                     1979   Alien 1979          0     
# ℹ 244 more rows
# ℹ Use `print(n = ...)` to see more rows
```
:::

We have several perfect matches in the first 10 rows. These have a
distance of 0. There are some less-good-but-still-matches, such as A
Clockwork Orange where the year differs between IMDB and Plex. Then
there are a bunch of clear "not matched" movies, e.g. 12 Angry Men, 12
Years a Slave. We can see that a distance of 0.1 or more means the match
is not true.

Note that it says there are 244 more rows and shows us 10 (a total of
254 when we should have only 250). The 4 extra matches are duplicates
caused by a same-distance match to two different movies in the Plex
library. Let's get rid of them and then figure out our totals.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
# remove duplicate fuzzy match fails
match <- match[!duplicated(match[ , "titleyear.x"]), ]
# leaves 250 rows

# matches with distance of 0.1 or more are not a match
# the are the movies we want to look at
match <- match %>% 
  filter(dist >= 0.1)
# leaves 190 rows
```
:::

So I have 60 of the IMDB's Top 250 Movies. This is not very high. In my
defence, I am not a movie buff and my movie collection is not
particularly huge.

So what are those movies that I am missing? Let's sort them to be the
highest rated and figure out what I should add with some urgency!

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
match <- match[order(-match$Rating),]

# write file
lapply(match$titleyear.x, write, "Output/Data/imdb.txt", append=TRUE)
```
:::

<figure class="wp-block-table">
<table>
<tbody>
<tr class="odd">
<td>The Shawshank Redemption</td>
<td>1994</td>
<td>9.2</td>
</tr>
<tr class="even">
<td>12 Angry Men</td>
<td>1957</td>
<td>9</td>
</tr>
<tr class="odd">
<td>The Dark Knight</td>
<td>2008</td>
<td>9</td>
</tr>
<tr class="even">
<td>Schindler’s List</td>
<td>1993</td>
<td>8.9</td>
</tr>
<tr class="odd">
<td>The Good, the Bad and the Ugly</td>
<td>1966</td>
<td>8.8</td>
</tr>
<tr class="even">
<td>Fight Club</td>
<td>1999</td>
<td>8.7</td>
</tr>
<tr class="odd">
<td>Inception</td>
<td>2010</td>
<td>8.7</td>
</tr>
<tr class="even">
<td>Interstellar</td>
<td>2014</td>
<td>8.6</td>
</tr>
<tr class="odd">
<td>It’s a Wonderful Life</td>
<td>1946</td>
<td>8.6</td>
</tr>
<tr class="even">
<td>Life Is Beautiful</td>
<td>1997</td>
<td>8.6</td>
</tr>
</tbody>
</table>
<figcaption>The top 10 films I was missing…</figcaption>
</figure>

I have at least seen some of those films at some point in the past.

---

The post title comes from "Yet Another Movie" by Pink Floyd from "A
Momentary Lapse of Reason".


I’m not a big movie person. Nonetheless I have a media library with quite a few films in and I wondered how many “films to see before you die”-type movies I had in the collection, and how many were missing.
<strong>
 I used R to find the answers.
</strong>
I’ve described previously how to get a plain text dump of a Plex database using WebTools-NG. I did that for the Movies library of my Plex Media Server.

Yet Another Movie: IMDB Top 250 movies

This website is all about niche tech tips and this post is no exception.

I run a Plex media server. This allows me to stream my music collection
when I am out and about. Plex pass owners get the nifty plexamp app for
listening to music, which I really like. The databasing for the movie
and TV show side of Plex works great, but the music side has its
peculiarities.

## The problem {#the-problem .wp-block-heading}

I have a large music library. It is meticulously curated with full
metadata in iTunes/Music style. Plex can handle this but there are a
couple of foibles that don't work so well and I noticed that my music
library had developed a few glitches. The major one being that about 70%
of my compilation albums had all been given the artist name of **Liz
Phair with Material Issue**! They covered the Tra La La theme from the
Banana Splits show and due to some conflict or other, that artist name
propagated to almost all compilations.

The problem is that compilations can be checked as compilations in
iTunes/Music yet have their Album Artist metadata blank, and be
understood to be a compilation. Plex doesn't understand this and treats
each song as a one track album for each artist on the compilation. For
some reason this led to the unusual naming issue above.

There were some other problems with Album Artist names that I won't bore
you with.

## The fix {#the-fix .wp-block-heading}

So the first thing was to flush the "Album Artist" name for all
Compilations with *Various Artists*. Which can be done easily in
iTunes/Music. However,

- A rescan of the Music library after this change did not fix the
  database.
- Doing the Plex Dance did not work either. This is where you move the
  offending media out of the folder that Plex watches, scan, clean
  bundles, and then move back and rescan.
- Drastic action was need.

This meant I needed to make a new Music Library.

Given the emotional attachment I have to my library and play history and
so on, I couldn't bring myself to delete the music library and remake
it. Instead I chose to make a new library, verify it was an improvement
and then either delete and replace the old library, or fix the problems
in the old library.

## How I did it {#how-i-did-it .wp-block-heading}

I first made a new music library (I called it `musica` for a temporary
name) with all the same settings as the music library. Creating this
library took a long time -- I have a lot of music!!

Next the problem was how to check the libraries. There have been various
plug-ins and scripts developed over the years (plex2csv, ExportTools)
but the current approach (seemingly endorsed by Plex) is this:
[WebTools-NG](https://github.com/WebTools-NG/WebTools-NG). Install was
simple on my mac and it was quick to log in to my server.

WebTools-NG can export each Plex library as a delimited text file. I
took no chances and just asked it to dump out everything for both
libraries. Each export took around 4 hours. In retrospect I could have
selected only the basic information and it would have been faster.

## Into R {#into-r .wp-block-heading}

I read the files into R. The files use pipe as a separator. The data
frames are huge and have lots of irrelevant columns (69 in total), so I
reduced them to the information that matters like this.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
music <- read.delim("Data/music.csv",sep = "|")
musica <- read.delim("Data/musica.csv",sep = "|")

reduce_library <- function(input) {
  output <- subset(input, select = c("Album.Title",
                           "Album.Year",
                           "Title",
                           "Audio.Track.Album.No",
                           "Audio.Track.No",
                           "Audio.Track.Artist",
                           "Duration",
                           "Part.File.Combined",
                           "Part.Size.as.Bytes"))
  return(output)
}

music <- reduce_library(music)
musica <- reduce_library(musica)
```
:::

I could see that I had 125389 rows in `music` and 125376 in `musica`. So
what was missing?

An anti-join will reveal the what is in music that is missing from
musica. We can use `Part.Files.Combined` which is a full file path to
each music file, to do these comparisons.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
library(dplyr)
only <- music %>%
  anti_join(musica,by = "Part.File.Combined")
```
:::

This revealed that one album (a 2-disc set) had not scanned properly.
But otherwise all other files were present in both libraries.

Next, the question was what other Artist Names were borked in `music`.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .r; .title: .; .notranslate title=""}
# merge music and musica to compare
all_df <- merge(x = music, y = musica, by = "Part.File.Combined", all = TRUE)
# find Artist conflicts
check <- all_df[with(all_df, Audio.Track.Artist.x != Audio.Track.Artist.y),]
# just look at artists' albums where there is a conflict rather than per track
check <- subset(check, select = c("Audio.Track.Artist.x","Audio.Track.Artist.y"))
check <- checks[!duplicated(check),]
```
:::

There were 4727 tracks with Artist conflicts and these were from 155
albums.

The problems were not just limited to compilations. Several other
artists had become misnamed in the original library. For many reasons, I
decided to delete the original library and then rename `musica` to
`music`. The only issue was then to reconnect all the apps and services
to the new library.

Weirdly, the play history seems to have carried over to the new library.
Which was a nice surprise since I had reconciled myself to losing this
info. One problem with the new library is that the waveform that can be
seen for each track when it plays in plexamp is missing until it can be
"scanned for loudness". This is a scheduled task and eventually they
will be replaced. You can check the logs to verify that there is
progress on this process.

## Conclusion {#conclusion .wp-block-heading}

If you find problems with your music library. Just go ahead and replace
it. There were very few errors in the remade library and it solved the
headaches I had with the original library.

---

The post title comes from "Multiplex" by Oliver from his album
"[Standing
Stone](https://www.discogs.com/master/339452-Oliver-Standing-Stone)".
One of those legendary unreleased/lost albums that got reissued in the
1990s. Recorded on reel-to-reel tape in a Welsh farmhouse, this album is
well worth seeking out.


This website is all about niche tech tips and this post is no exception. I run a Plex media server. This allows me to stream my music collection when I am out and about. Plex pass owners get the nifty plexamp app for listening to music, which I really like. The databasing for the movie and TV show side of Plex works great, but the music side has its peculiarities. The problem   I have a large music library.

Rogue Scholar Posts

Yet Another Movie: IMDB Top 250 movies

Multiplex: comparing Plex libraries