Scott Chamberlain

In February this year I wroute about how many parameters functions should have, looking at some other languages, with a detailed look at R. On a related topic …  As I work on many R packages that are API clients for various web services, I began wondering: What is the best way to deal with API routes that have a lot of parameters?

API client design: how to deal with lots of parameters?

notes to self for next job hunt (some of which may be generally useful): don’t apply to general tech companies anymore for many reasons. heard back from very very few but that may be b/c I don’t know many people at general tech companies never been able to get through interviews; they’re presumably looking for computer science grads (not me) most of their missions are probably not stuff I’d be happy about at the end of the day.

Job searching notes

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub. There were a few catches though.

Python, ast, and redbaron

My parents just found this email they had printed out from me from May 19, 2006, when I was 26. I chatted about some family stuff, then had this rambling string of weird thoughts below. I thought others might appreciate a good cringe - or cringy laugh - at my expense. It’s especially funny because I’m most def an atheist. I don’t know, those Tucson sunsets really are transformative.

Weird thoughts

I was working on a work task last week, and needed to filter out one instance of a class from a list of class instances. No matter how you do this speed doesn’t matter too much if you’re doing this operation once or a few times. However, I this operation needs to be done about 100K times each time the script runs - so speed definitely does matter in this case.

List comprehension vs. filter vs. key lookup

taxizedb arose from pain in using taxize when dealing with large amounts of data in a single request or doing a lot of requests of any data size. taxize works with remote data sources on the web, so there’s a number of issues that can slow the response down: internet speed, server response speed (was a response already cached or not; or do they even use caching), etc.

taxizedb: an update

I worked on a refactor of an R package at work the other day. Here’s some notes about that after doing the work. This IS NOT a best practices post - it’s just a collection of thoughts. For context, the package is an API client.

Refactoring notes

This blog is now using Hugo. Important - if you subscribe to the RSS for this blog you likely have to delete/remove the old one and add the new RSS link.

Moved to Hugo

I wrote the other day about overcoming an issue with Shiny. Another issue I ran into concurrently was about file inputs. The issue was that file inputs (i.e., shiny::fileInput) was difficult to clear.

Shiny file inputs

I recently opened an issue in a repository for a package I’m working on to think about potential footguns and how to avoid them. That word “footguns” got me thinking;

Avoiding the word footgun(s)

I’ve been working on an inherited Shiny app at work for the past few months. One of the many frustrating things about Shiny lately has been around buttons. Well, it wasn’t really about buttons, but that’s where it started. I noticed that a number of buttons - some having file inputs, some having text inputs - did not trigger every time, and I expected them to trigger every time no matter what.

Shiny button weirdness

TL;DR  In 6 months (end of November 2022) the CRAN Checks API https://cranchecks.info/ will be gone You can still get badges at https://badges.cranchecks.info You can use the new badges like: [![cran checks](https://badges.cranchecks.info/worst/dplyr.svg)](https://cran.r-project.org/web/checks/check_results_dplyr.html)  Find more details at https://github.com/sckott/cchecksbadges Sunsetting the CRAN Checks API   If you contribute an R package to

CRAN Checks API and Badges

It’s been interesting switching jobs with respect to programming languages. I used to write 95% R - now I write 95% Python.  I have been using Python for many years, but not seriously or getting paid either. I’ve learned alot in the first 6 months.  Some Python things learned: Functions and methods   I used to think functions and methods were the same thing. But during the last 6 months I learned that functions and methods are not the same.

Notes on Python

Soooo, my last job at Deck was amazing. I loved it. I was doing data engineer stuff there, mostly maintaining infrastructure for data pipelines. Everyone was great and the mission was amazing: helping Democrats win. Yet the company was shut down about a month ago, sending me on another job search, the 3rd since early/mid 2021.

Working at Fred Hutchinson Cancer Center

You’ve experienced an HTTP redirect (or URL redirect, or URL forwarding) even if you haven’t noticed. We all use browsers (I assume, since you are reading this), either on a phone or laptop/desktop computer. Browsers don’t show all the HTTP requests going on in the background, some of which are redirects.

Mocking HTTP redirects

\  Code is often arranged in packages for any given language. Packages are often cataloged in a package registry of some kind: NPM for node, crates.io for Rust, etc. For R, that registry is either CRAN or Bioconductor (for the most part).  CRAN has the concept of an archived package.

stories behind archived packages

Recology

I recently had a use case at work where I wanted to check that file paths given in a Python script actually existed. These paths were in various GitHub repositories, so all I had to do was pull out the paths and check if they exist on GitHub.

There were a few catches though.

First, I couldn't simply get any string out of each Python script - they needed to be strings specficied by a specific function parameter, and match a regex (e.g., start with 'abc').

Second, the script paths lack the GitHub repository root name. This name was part of the function name - so I needed to get access to the function that the path was specified within, and then parse the function name to get the repository name.

The obvious solution I thought was the [ast](https://docs.python.org/3/library/ast.html) library.

## ast library

I started by using `ast`{.language-plaintext .highlighter-rouge}. The `ast.NodeVisitor`{.language-plaintext .highlighter-rouge} class seemed like it would do the trick.

An example script ("my_script.py"):

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
def hello(path, stuff=None):
    return path


if __name__ == "__main__":
    print(hello(path="hello/world.py", stuff="hello mars"))
```
:::
:::

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
import ast

class CollectStrings(ast.NodeVisitor):
    def visit_Module(self, node):
        self.out = set()
        self.generic_visit(node)
        return list(filter(lambda w: w.startswith("hello") and w.endswith(".py"), self.out))

    def visit_Str(self, node):
        self.out.add(node.s)

file = "my_script.py"
with open(file, "r") as f:
    body = ast.parse(f.read())

coll = CollectStrings()
coll.visit(body)
## ['hello/world.py']
```
:::
:::

That worked great at fetching paths - only because all the paths I was looking for started with the same text and all have the same file extension.

HOWEVER - I also needed the function name that the `path`{.language-plaintext .highlighter-rouge} argument was called from. I tried to make this work with `ast.NodeVisitor`{.language-plaintext .highlighter-rouge} but couldn't get it to work.

I eventually got frustrated enough and figured there must be some libraries that build on top of `ast`{.language-plaintext .highlighter-rouge} that make it easier to work with ast's in Python.

## redbaron

Enter [redbaron](https://github.com/PyCQA/redbaron). I found this library pretty quickly upon searching for a library building on top of `ast`{.language-plaintext .highlighter-rouge}.

Another example script ("their_script.py"):

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
def hello(path, stuff=None):
    return path


def goodbye(path, stuff=None):
    return path


def world():
    path_str = hello(path="src/world.py", stuff="hello mars")
    other_path_str = goodbye(path="src/world.py", stuff="hello saturn")

    return path_str, other_path_str


if __name__ == "__main__":
    print(world())
```
:::
:::

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
import re
from redbaron import RedBaron

file = "their_script.py"
with open(file, "r") as src:
  red = RedBaron(src.read())

red
## 0   def hello(path, stuff=None):
##         return path
##     
##     
##     
## 1   def goodbye(path, stuff=None):
##         return path
##     
##     
##     
## 2   def world():
##         path_str = hello(path="src/world.py", stuff="hello mars")
##         other_path_str = goodbye(path="src/world.py", stuff="hello saturn")
##     
##         return path_str, other_path_str
##     
##     
##     
## 3   if __name__ == "__main__":
##         print(world())
## 
```
:::
:::

Even just the resulting object you get from parsing something is useful:

And with `.help()`{.language-plaintext .highlighter-rouge} you get a very detailed map of the structure of the thing you're trying to navigate (only printing first 20 lines):

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
red.help()
## 0 -----------------------------------------------------
## DefNode()
##   # identifiers: def, def_, defnode, funcdef, funcdef_
##   # default test value: name
##   async=False
##   name='hello'
##   return_annotation ->
##     None
##   decorators ->
##   arguments ->
##     * DefArgumentNode()
##         # identifiers: def_argument, def_argument_, defargument, defargumentnode
##         target ->
##           NameNode() ...
##         annotation ->
##           None
##         value ->
##           None
##     * DefArgumentNode()
##         # identifiers: def_argument, def_argument_, defargument, defargumentnode
...
```
:::
:::

Looking at the result from `red.help()`{.language-plaintext .highlighter-rouge} I can then use `.find_all()`{.language-plaintext .highlighter-rouge} to find certain nodes in the ast.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
nodes = red.find_all("AtomtrailersNode")
nodes = list(filter(lambda w: "hello" in w.dumps(), nodes))
nodes
## [hello(path="src/world.py", stuff="hello mars"), goodbye(path="src/world.py", stuff="hello saturn")]
```
:::
:::

Then I can write some okay code to extract out the function name, and ugly code to get the string supplied to the `path`{.language-plaintext .highlighter-rouge} parameter. Then f-string those together to get the path I'm after.

::: {.language-python .highlighter-rouge}
::: highlight
``` highlight
paths = []
for node in nodes:
    fxn_name = node.name.value
    command = re.search("src/.*\\.py", node.dumps()).group()
    paths.append(f"{fxn_name}/{command}")

for path in paths:
    print(path)
## hello/src/world.py
## goodbye/src/world.py
```
:::
:::

Not super proud of this but gets the job done for my use case - and when you're not making open source for others, you don't need to worry about other use cases :)

I'll definitely try to learn how to properly extract stuff using `redbaron`{.language-plaintext .highlighter-rouge} - but it got me to answer much faster than the `ast`{.language-plaintext .highlighter-rouge} library.
