Stephen Royle

I wrote previously about how to [archive a Slack
workspace](https://quantixed.org/2022/08/11/slacker-archiving-information-from-a-slack-workspace/){data-type="post"
data-id="2793"}.

Well, now Slack announced a new policy, effective from August 26th,
2024. Under the new policy, messages and files older than one year in
workspaces on the free tier will be **deleted**. This is in addition to
the 90 day limit for viewing content in the workspace.

This causes a problem with the solution I outlined previously. Briefly,
the viewable archive generated by the excellent
[slack-export-viewer](https://github.com/hfaran/slack-export-viewer) is
served up using a flask application. However, all the content (images
and links to files) are all hosted by Slack. This means that
**everything will disappear when slack deletes any files older than one
year**.

## Time for action! {#time-for-action .wp-block-heading}

**tl;dr** I have written a solution, which is available
[here](https://github.com/quantixed/slack-export-local).

For anyone who is interested in the details, read on.

## More info {#more-info .wp-block-heading}

The problem is really contained to stuff hosted by slack, every other
external link should be OK (well they're susceptible to link-rot, but
let's assume we can live with that). There are three types of files:
avatars (nice to have), previews of the files (thumbnails) and the files
themselves. If we assume all three types will disappear, we need to
rescue them!

The avatars are hosted like this:

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
https://avatars.slack-edge.com/YYY-MM-DD/13NumericChars_20HexChars_res.jpg
```
:::

There are also gravatars for people who don't have a hosted avatar.
These are hosted at `secure.gravatar.com`

Files are hosted like this:

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
https://files.slack.com/files-pri/teamid-userid/filename?t=secret_token
```
:::

The thumbnails are hosted in a similar way in a directory called
`files-tmb` rather than `files-pri` and with the size of the thumbnail
appended to the filename.

What is surprising is that you can access all your old files (as long as
they weren't manually deleted on Slack) going back to the beginning of
the workspace. Even if you are on the free tier and the files are older
than 90 days. Note that this is currently the case, and obviously once
Slack start to delete stuff you won't have access at all! This is all
possible via a secret token which is essential for viewing the content
and for downloading the files. If you exported an archive some time age,
it is possible that the token will expire and you will lose access to
your files.

You can see all of this by looking at the json files in an exported
archive. What `slack-export-viewer` does is to build all of that into a
viewable archive. I was able to write something quickly which downloaded
all the content, but I couldn't figure out how to get it to display in
the flask application...

The nice thing is that `slack-export-viewer` can also generate a local
static html instance of the archive using:

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .bash; .title: .; .notranslate title=""}
slack-export-viewer -z path/to/archive.zip --html-only
```
:::

OK, now we can work with this static html instance. We just need to
download all of the linked content, store it locally in a way that can
be accessed easily by the instance and then change all the links to
point to the local instance. This is what the python script does.

It would be really cool to add this option into `slack-export-viewer`
itself, but it is beyond me to do that and generate a PR.

Here's the code as it stands. I'm not a pythonista, so if anyone has any
suggestions for improvement, hmu. Up-to-date version is on GitHub
[here](https://github.com/quantixed/slack-export-local).

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .python; .title: .; .notranslate title=""}
#!/usr/bin/python

import sys
import os
import requests
from urllib.parse import urlparse
from bs4 import BeautifulSoup

def download_file(url, root_dir, html_file_path):
    """Download a file from a URL into the specified directory structure if it matches allowed domains."""
    allowed_domains = ['files.slack.com', 'avatars.slack-edge.com', 'secure.gravatar.com']
    parsed_url = urlparse(url)
    
    # Check if the URL's domain is in the list of allowed domains
    if parsed_url.netloc not in allowed_domains:
        return url  # Return the original URL if it's not from an allowed domain
    
    # Create a path based on the URL structure
    file_path = os.path.join(root_dir, parsed_url.netloc, parsed_url.path.strip("/"))
    os.makedirs(os.path.dirname(file_path), exist_ok=True)

    # Creat the relative path to the html file
    file_path_relative = os.path.relpath(file_path, os.path.dirname(html_file_path))

    # Skip downloading if the file already exists
    if os.path.exists(file_path):
        return file_path_relative

    # Download and save the file
    response = requests.get(url)
    with open(file_path, 'wb') as file:
        file.write(response.content)
    
    return file_path_relative

def update_html_file(html_file_path, root_dir):
    """Update HTML file to download and reference local copies of external resources."""
    with open(html_file_path, 'r', encoding='utf-8') as file:
        soup = BeautifulSoup(file, 'html.parser')

    # Handle img tags with src attributes
    for img_tag in soup.find_all('img'):
        original_url = img_tag.get('src')
        if original_url:
            new_path = download_file(original_url, root_dir, html_file_path)
            img_tag['src'] = new_path
    
    # Handle tags with href attributes (e.g., a, link)
    for href_tag in soup.find_all(href=True):
        original_url = href_tag.get('href')
        if original_url:
            new_path = download_file(original_url, root_dir, html_file_path)
            href_tag['href'] = new_path
    
    # Write the updated HTML back to a file
    with open(html_file_path, 'w', encoding='utf-8') as file:
        file.write(str(soup))

def scan_and_update_html_files(channel_dir, root_dir):
    """Scan each folder in the channel directory and update the index.html file found within."""
    for subdir, dirs, files in os.walk(channel_dir):
        # Print message to indicate the current directory being processed
        print(f"Processing directory: {subdir}")
        for file in files:
            if file == 'index.html':
                html_file_path = os.path.join(subdir, file)
                update_html_file(html_file_path, root_dir)

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python slack-export-local.py <root_directory_path>")
        sys.exit(1)

    root_directory_path = sys.argv[1]
    channel_dir = os.path.join(root_directory_path, "channel")
    root_dir = os.path.join(root_directory_path, "attachments")

    scan_and_update_html_files(channel_dir, root_dir)
```
:::

---

The post title comes from "Slacker" by Wall of Sleep off
their [eponymous
album](https://www.discogs.com/master/460072-Wall-Of-Sleep-Wall-Of-Sleep).
So tempted to name this post after the famous Superchunk song, but this
is a family-friendly website.


I wrote previously about how to archive a Slack workspace. Well, now Slack announced a new policy, effective from August 26th, 2024. Under the new policy, messages and files older than one year in workspaces on the free tier will be
<strong>
 deleted
</strong>
. This is in addition to the 90 day limit for viewing content in the workspace. This causes a problem with the solution I outlined previously.

Slacker II: archiving information from a Slack workspace

Slack is a wonderful tool for communicating and organising work in
teams. Many academic groups are using the free version (since the
pricing is simply incompatible with how we are funded). Slack has had a
10,000 message limit for free tier workspaces for many years but they
recently changed it to a 90 day limit. This means any messages older
than 90 days will no longer be visible.

The way we use our lab slack workspace is mainly for rapid
communication. What's there doesn't really need to be archived, but
there are occasional posts in the public channels which are useful.
**How can we archive this important information?**

Unsurprisingly there are many solutions to this problem. I will describe
one approach that worked for us. Feel free to comment if you know
something better.

I have been exporting our slack workspace data at approximately 90 day
intervals for some time, so I have several zipped archives of our data.
If you don't have this, and the 90 day limit has not kicked in or you
find yourself gifted with a free trial of pro (giving you access to
everything), download your data **immediately**. Failing this, my advice
would be to just download your last 90 days and start archiving!

The archive is a bunch of json files. You will have folders for each
channel (it is not possible for you to download DMs) in each will be
many json files (one for each day that there was activity). These files
are not really human readable. So to view the data I used
`slack-export-viewer` available
[here](https://github.com/hfaran/slack-export-viewer).

## Getting going {#getting-going .wp-block-heading}

I used `pipx` for install as suggested. I had to install `pipx` on my
machine via `homebrew`

I am running macOS 12.5 and I found that the launch of `localhost:5000`
in my browser did not work at first. I had to edit my `/etc/hosts` file
which is possible using `sudo nano /private/etc/hosts` simply adding

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
127.0.0.1       localhost
```
:::

to the file and saving. For those that don't know, `sudo` will require
an admin password and `nano` can be saved and closed using ctrl+x and
saying yes.

## All good, now to view the slack data. {#all-good-now-to-view-the-slack-data. .wp-block-heading}

If you have multiple zipped archives, they can each be extracted and,
using `rsync`, combined into one single folder. Maybe you just want to
have a look at one archive first though. `slack-export-viewer` will open
a zip archive directly or it will take an extracted archive.

::: {.wp-block-syntaxhighlighter-code}
``` {.brush: .plain; .title: .; .notranslate title=""}
slack-export-viewer -z path/to/archive
```
:::

And amazingly your slack archive pops up in your browser!

<figure class="wp-block-image size-large">
<img
src="https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?resize=640%2C361&amp;ssl=1"
class="wp-image-2794" loading="lazy" decoding="async"
data-attachment-id="2794"
data-permalink="https://quantixed.org/2022/08/11/slacker-archiving-information-from-a-slack-workspace/screenshot-2022-08-11-at-07-46-11/"
data-orig-file="https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?fit=1611%2C909&amp;ssl=1"
data-orig-size="1611,909" data-comments-opened="1"
data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}"
data-image-title="Screenshot-2022-08-11-at-07.46.11"
data-image-description="" data-image-caption=""
data-medium-file="https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?fit=300%2C169&amp;ssl=1"
data-large-file="https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?fit=640%2C361&amp;ssl=1"
srcset="https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?resize=1024%2C578&amp;ssl=1 1024w, https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?resize=300%2C169&amp;ssl=1 300w, https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?resize=768%2C433&amp;ssl=1 768w, https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?resize=1536%2C867&amp;ssl=1 1536w, https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?w=1611&amp;ssl=1 1611w, https://i0.wp.com/quantixed.org/wp-content/uploads/2022/08/Screenshot-2022-08-11-at-07.46.11.png?w=1280&amp;ssl=1 1280w"
sizes="(max-width: 640px) 100vw, 640px" data-recalc-dims="1" width="640"
height="361" />
</figure>

The formatting is pretty good and it's certainly good enough to scroll
around and copy out any useful information. I went through our
microscopy channel and picked out the posts that folks had made that we
wanted to archive and copied them direct to a post on our ELN. I'm sure
it's possible to serve this archive up as a "website" but the majority
of the info in there is not really useful for us to put in the effort
for this.

**The best bit is that all the attachments are available and can be
downloaded direct from slack with no restriction.** In the screenshot
above you can see two pictures, they can be clicked on and downloaded at
full resolution. PDFs could be pulled down too.

## Final word {#final-word .wp-block-heading}

This tip worked for our use case and it might work for you too. Just to
reiterate that there are *lots* of slack tools out there and I didn't do
much investigating since I had the archives and just wanted to extract
some info.

---

The post title comes from "Slacker" by Wall of Sleep off their
[eponymous
album](https://www.discogs.com/master/460072-Wall-Of-Sleep-Wall-Of-Sleep).


Slack is a wonderful tool for communicating and organising work in teams. Many academic groups are using the free version (since the pricing is simply incompatible with how we are funded). Slack has had a 10,000 message limit for free tier workspaces for many years but they recently changed it to a 90 day limit. This means any messages older than 90 days will no longer be visible.

Rogue Scholar Beiträge

Slacker II: archiving information from a Slack workspace

Slacker: archiving information from a Slack workspace