Albert Krewinkel

One of the many great things about free and open source software and the
whole GNU/Linux ecosystem are the simple yet powerful tools available.
The possibilities enabled by almost trivial programs are incredible. A
very positive side effect this has on me is that I like to go and
explore technologies with the tools at my disposal. My latest
experiments revolved around the HTTP protocol, specifically HTTP
headers, and very basic open source networking tools.

::: {#the-hypertext-transfer-protocol .section .level2}
## The HyperText Transfer Protocol {.anchored anchor-id="the-hypertext-transfer-protocol"}

Webservers on the internet sending a website to a browser use the
HyperText Transfer Protocol (HTTP) to do so. Along with the HTML data
for the page itself, the server answer includes additional information:
Response code, cookies, and how the browser or proxy server should
handle the contents is transfered within the *HTTP header*. The ability
of the headers to control state on the client side is what makes them so
interesting and the reason why we are going to have a closer look at
them.
:::

::: {#communicating-with-netcat .section .level2}
## Communicating with `netcat` {.anchored anchor-id="communicating-with-netcat"}

Instead of building our own HTTP client and server implementations --
that would be total overkill -- we restrict the goal to a simple
networking tool that can be made to receive, send and alter basic HTTP
commands: `netcat`, the self-described TCP/IP swiss army knife, combined
with basic shell scripts.

We start by setting up a basic echoing server which sends everything
back the same way it was received.

``` shell
nc -l -p 8042 -e '/bin/cat'
```

Pointing the browser at `http://localhost:8042`, then killing the
`netcat` process manually by hitting `Control-C`, we can see the headers
we sent within our browser. Everything that is send to our simple server
is put through the `cat` program, which just passes it on to STDOUT,
which is then sent back to the connecting browser. The process has to be
terminated manually, as it doesn't know when to stop listening for more
input. It's crude, yet effective.
:::

::: {#shell-scripting-for-more-advanced-features .section .level2}
## Shell Scripting for more advanced features {.anchored anchor-id="shell-scripting-for-more-advanced-features"}

The above is neither comfortable to use nor very good to toy with, so we
replace `cat` with a script of our own, we'll call it
`exploring-http.sh`:

``` shell
#!/bin/sh

# Keep reading everything until we hit the first empty line
read_headers ()
{
    read i
    while [ -n "$i" ] &&
          [ "$(echo -n "\r\n")" != "$i" ] &&
          [ "$(echo -n "\n")" != "$i" ]
    do
        echo "$i"
        read i
    done
}
request_headers="$(read_headers)"

# Get some response headers ready
response_headers ()
{
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "\r\n"
}

# Send the response
respond ()
{
    local response_headers="$(response_headers)"
    echo "${response_headers}"
    echo "Browser Request Headers"
    echo "======================="
    echo "$request_headers"
    echo "\r\n"
    echo "Server Response Headers"
    echo "======================="
    echo "${response_headers}"
}

respond
```

The request send by the browser is read till we reach the first black
line, signaling the end of the request header. This time, we follow the
protocol by prefixing the content with very simple response headers
before sending it back to the browser. We also don't have to manually
terminate our `netcat` server, it terminates after answering to the
request. Starting it again after each request is tedious, so we automate
it and put it into a loop, restarting the server immediately once it
terminates.

``` shell
sh -c 'while true; do nc -l -p 8042 -e exploring-http.sh; done'
```

Now we are free to experiment with HTTP headers and the way browser and
server interact. For example, we can let the server add a
`Last-Modified` header, the content of which should be sent back by the
browser in the next request:

``` shell
response_headers ()
{
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "Last-Modified: $(date --rfc-2822)\r\n"
    printf "\r\n"
}
```

Reloading twice, and the browser request will change to send an
additional `If-Modified-Since` header.
:::

::: {#etags .section .level2}
## ETags {.anchored anchor-id="etags"}

The functionality of ETags, designed to communicate caching of old
files, can be used follow users around without the need of cookies.
Let's see if we can do this with our little server.

The function generating the response headers is modified to extract any
ETag supplied by the browser. If none exists, we generate a new one by
hashing the number of nanoseconds passed since the beginning of the UNIX
epoche. The parsed or newly generated etag is then sent back to the
browser. We also add a few header to make sure the conents isn't cached.
As a result, we should be able to track a user through his or her
browser cache.

``` shell
response_headers ()
{
    local etag
    etag=$(echo "${request_headers}" | sed -ne 's/^$If-None-Match: "\([a-f0-9]*$".*\)/\2/gp')
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "Last-Modified: $(date --rfc-2822)\r\n"
    printf "ETag: \"${etag:-$(date +%s%N | md5sum | cut -d' ' -f1)}\"\r\n"
    printf "Expires: Tue, 01 Jan 2013 00:00:01 GMT\r\n"
    printf "Cache-Control: max-age=0\r\n"
    printf "Connection: keep-alive\r\n"
    printf "\r\n"
}
```

We can test this by reloading our test page twice and... it works! We
can reload as often as we want, the ETag header sent by the browser will
not change unless we clear the browser's cache. A stealthy kind of user
tracking can be simulated with just a few lines of shell script.
:::

::: {#conclusion .section .level2}
## Conclusion {.anchored anchor-id="conclusion"}

Even though we used nothing but simple command line tools and shell
scripting, we managed to build a simple server and to experiment with
the ways in which stateless servers and stateful browseres can effect
each other through HTTP headers. Standard UNIX tools are very powerful
by themselves; together with a tool like `netcat`, power and fun extend
even into experiments with networking and default protocols.
:::


One of the many great things about free and open source software and the whole GNU/Linux ecosystem are the simple yet powerful tools available. The possibilities enabled by almost trivial programs are incredible. A very positive side effect this has on me is that I like to go and explore technologies with the tools at my disposal.

Exploring HTTP Headers with netcat

<p>One of the many great things about free and open source software and the whole GNU/Linux ecosystem are the simple yet powerful tools available. The possibilities enabled by almost trivial programs are incredible. A very positive side effect this has on me is that I like to go and explore technologies with the tools at my disposal. My latest experiments revolved around the HTTP protocol, specifically HTTP headers, and very basic open source networking tools.</p>
<section id="the-hypertext-transfer-protocol" class="level2">
<h2 class="anchored" data-anchor-id="the-hypertext-transfer-protocol">The HyperText Transfer Protocol</h2>
<p>Webservers on the internet sending a website to a browser use the HyperText Transfer Protocol (HTTP) to do so. Along with the HTML data for the page itself, the server answer includes additional information: Response code, cookies, and how the browser or proxy server should handle the contents is transfered within the <em>HTTP header</em>. The ability of the headers to control state on the client side is what makes them so interesting and the reason why we are going to have a closer look at them.</p>
</section>
<section id="communicating-with-netcat" class="level2">
<h2 class="anchored" data-anchor-id="communicating-with-netcat">Communicating with <code>netcat</code></h2>
<p>Instead of building our own HTTP client and server implementations – that would be total overkill – we restrict the goal to a simple networking tool that can be made to receive, send and alter basic HTTP commands: <code>netcat</code>, the self-described TCP/IP swiss army knife, combined with basic shell scripts.</p>
<p>We start by setting up a basic echoing server which sends everything back the same way it was received.</p>
<pre class="shell"><code>nc -l -p 8042 -e '/bin/cat'</code></pre>
<p>Pointing the browser at <code>http://localhost:8042</code>, then killing the <code>netcat</code> process manually by hitting <code>Control-C</code>, we can see the headers we sent within our browser. Everything that is send to our simple server is put through the <code>cat</code> program, which just passes it on to STDOUT, which is then sent back to the connecting browser. The process has to be terminated manually, as it doesn’t know when to stop listening for more input. It’s crude, yet effective.</p>
</section>
<section id="shell-scripting-for-more-advanced-features" class="level2">
<h2 class="anchored" data-anchor-id="shell-scripting-for-more-advanced-features">Shell Scripting for more advanced features</h2>
<p>The above is neither comfortable to use nor very good to toy with, so we replace <code>cat</code> with a script of our own, we’ll call it <code>exploring-http.sh</code>:</p>
<pre class="shell"><code>#!/bin/sh

# Keep reading everything until we hit the first empty line
read_headers ()
{
    read i
    while [ -n "$i" ] &amp;&amp;
          [ "$(echo -n "\r\n")" != "$i" ] &amp;&amp;
          [ "$(echo -n "\n")" != "$i" ]
    do
        echo "$i"
        read i
    done
}
request_headers="$(read_headers)"

# Get some response headers ready
response_headers ()
{
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "\r\n"
}

# Send the response
respond ()
{
    local response_headers="$(response_headers)"
    echo "${response_headers}"
    echo "Browser Request Headers"
    echo "======================="
    echo "$request_headers"
    echo "\r\n"
    echo "Server Response Headers"
    echo "======================="
    echo "${response_headers}"
}

respond</code></pre>
<p>The request send by the browser is read till we reach the first black line, signaling the end of the request header. This time, we follow the protocol by prefixing the content with very simple response headers before sending it back to the browser. We also don’t have to manually terminate our <code>netcat</code> server, it terminates after answering to the request. Starting it again after each request is tedious, so we automate it and put it into a loop, restarting the server immediately once it terminates.</p>
<pre class="shell"><code>sh -c 'while true; do nc -l -p 8042 -e exploring-http.sh; done'</code></pre>
<p>Now we are free to experiment with HTTP headers and the way browser and server interact. For example, we can let the server add a <code>Last-Modified</code> header, the content of which should be sent back by the browser in the next request:</p>
<pre class="shell"><code>response_headers ()
{
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "Last-Modified: $(date --rfc-2822)\r\n"
    printf "\r\n"
}</code></pre>
<p>Reloading twice, and the browser request will change to send an additional <code>If-Modified-Since</code> header.</p>
</section>
<section id="etags" class="level2">
<h2 class="anchored" data-anchor-id="etags">ETags</h2>
<p>The functionality of ETags, designed to communicate caching of old files, can be used follow users around without the need of cookies. Let’s see if we can do this with our little server.</p>
<p>The function generating the response headers is modified to extract any ETag supplied by the browser. If none exists, we generate a new one by hashing the number of nanoseconds passed since the beginning of the UNIX epoche. The parsed or newly generated etag is then sent back to the browser. We also add a few header to make sure the conents isn’t cached. As a result, we should be able to track a user through his or her browser cache.</p>
<pre class="shell"><code>response_headers ()
{
    local etag
    etag=$(echo "${request_headers}" | sed -ne 's/^$If-None-Match: "\([a-f0-9]*$".*\)/\2/gp')
    printf "HTTP/1.1 200 OK\r\n"
    printf "Content-Type: text/plain\r\n"
    printf "Last-Modified: $(date --rfc-2822)\r\n"
    printf "ETag: \"${etag:-$(date +%s%N | md5sum | cut -d' ' -f1)}\"\r\n"
    printf "Expires: Tue, 01 Jan 2013 00:00:01 GMT\r\n"
    printf "Cache-Control: max-age=0\r\n"
    printf "Connection: keep-alive\r\n"
    printf "\r\n"
}</code></pre>
<p>We can test this by reloading our test page twice and… it works! We can reload as often as we want, the ETag header sent by the browser will not change unless we clear the browser’s cache. A stealthy kind of user tracking can be simulated with just a few lines of shell script.</p>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>Even though we used nothing but simple command line tools and shell scripting, we managed to build a simple server and to experiment with the ways in which stateless servers and stateful browseres can effect each other through HTTP headers. Standard UNIX tools are very powerful by themselves; together with a tool like <code>netcat</code>, power and fun extend even into experiments with networking and default protocols.</p>


</section>

Cameron Neylon

::: {.wp-caption .alignleft style="width: 310px"}
[![English: Passport entry stamp for citizens
and\...](http://cameronneylon.net/wp-content/uploads/2012/02/300px-Malaysia_klia_entry_citizen2.jpg "English: Passport entry stamp for citizens and..."){.zemanta-img-inserted
.zemanta-img-configured decoding="async" loading="lazy" width="300"
height="188"}](http://commons.wikipedia.org/wiki/File:Malaysia_klia_entry_citizen.JPG)

Image via Wikipedia
:::

A few weeks ago I attended a workshop run by the [ESRC Genomics
Forum](http://www.genomicsnetwork.ac.uk/forum/) in Edinburgh which
brought together humanists, social scientists, and science focused folks
with an interest in how open approaches can and should be applied to
genomic science. This was interesting on a number of levels but I was
especially interested in the comments of [Marina
Levina](http://www.marinalevina.com/) on citizenship. In particular she
asked the question â€œwhat are the civic responsibilities of a network
citizen?â€?

Actually she asked me this question several times and it took me until
quite late in the day to really understand what she meant. I initially
answered with reference to [Clay Shirky](http://www.shirky.com/) on the
rise of creative contribution on the web as if just making stuff was all
that a citizen need do but what Marina was getting at was a deeper
question about a shared sense of responsibilities.

[Citizenship as a concept is a vexed
question](http://en.wikipedia.org/wiki/Citizenship) and there are a
range of somewhat incompatible philosophical approaches to describing
and understanding it. For my purposes here I want to focus on
citizenship as a sense of belonging to a group with shared values and
resources, and rights to access those resources. Traditionally these
allegiances lie with the nation state but, while nationalism is
undeniably on the rise, there seems to be a growing group of us who have
a patchwork of citizenships with different groups and communities.

Many of these communities live on the web and benefit from the use of
the internet as a sort of commons. At the same time there has been a
growing sense of behavioural norms and responsibilities in some parts of
the social web: a sophisticated sense of identity, the responsibility to
mark spam for takedown, a dedication to broad freedom of expression,
perhaps even a growing understanding of the tensions between that
freedom and â€œciviltyâ€?.

In the context of research on the web we have often talked about the
value of â€œnormsâ€? of behaviour as a far better mechanism for
regulation than licences and legal documents. A sense of belonging to a
community, of being a citizen, and the consequent risk of exclusion for
bad behaviour is a powerful encouragement to adhere to those norms, even
if that exclusion is just being shunned. Of course such enforcement can
lead to negative consequences as well as positive but I would argue that
in our day to day activities in most cases an element of social pressure
has a largely positive effect.

A citizen has a responsibility to contribute to the shared resources
that support the community. In a nation state we pay taxes, undertake
jury duty, vote in elections. What are the contributions expected of a
network citizen? Taking one step back, what are those shared resources?
The internet and the underlying framework of the web are one set of
resources. Of course these are resources that lie at the intersection of
our traditional states, as physical and commercial resources, and our
network society. In this context the protests against SOPA, PIPA, and
ACTA might be seen as the citizens of the network attending a rally,
perhaps even mobilizing our â€œmilitaryâ€? if only to demonstrate their
capacity.

But the core resources of the network are the nodes on the network and
the connections between them. The people, information resources, and
tools make up the nodes, and the links connecting them are what actually
makes them usable. As citizens of the network our contribution is to
make these links, to tend the garden of resources, to build tools. Above
all our civic duty is to *share*.

It is a commonly made point that with digital resources being infinitely
copyable there is no need for a tragedy of the commons. But there is a
flip side to this â€" when we think of physical commons we often think
of resources that donâ€™t need active maintenance. As long as they are
properly managed, not over-grazed or polluted, there is a sense that
these physical commons will be ok. The digital commons requires constant
maintenance. As an information resource it needs to be brought up to
date. And with these constant updates the tools and resources need to be
constantly checked for interoperability.

Maintaining these resources requires work. It requires money and it
requires time. The active network citizen contributes to these
resources, modifying content, adding links, removing vandalism. In
exchange for this the active network citizen obtains influence â€" not
dissimilar to getting to vote in elections â€" in those discussions
about norms and behaviour. But the core civic duty is to share, with the
expectation that other citizens, in their turn, will share back; that
working together as a community the citizenry will build, maintain, and
strengthen the civic institutions of the network.

This analysis scales beyond individual people to organizations.
Wikipedia is an important civic institution of network, one that accepts
a tithe from the active citizen in the form of time and eyeballs but
which gives much back to the community in the form of links and high
quality resources. Google accepts the links we make and gives back
search results but isnâ€™t always quite such a good citizen, breaking
standards, removing the RSS feeds that could be used by others.
Facebook? Well the less said the better. But good citizens will both
take what they need from the pool of resources and contribute
effectively back to the common institutions, those aggregation points
for resources and tools that make the network an attractive place to
live and work.

And I use â€œworkâ€? advisedly because a core piece of the value of the
network is the ability for citizens to use it to do their jobs, for it
to be a source of resources tools and expertise, that can be used by
people to make a living. And the quid pro quo is that the good citizen
contributes back resources that others might use to make money. In a
viable community with a viable commons there will be money, or its
equivalent, being generated and spent. A networked community will
encourage its citizens to generate value because this floats all boats
higher. In return for taking value out of the system the good citizen
will contribute it back. But they will do this as a matter of principle,
as part of their social contract, not because a legal document tells
them to. Indeed requiring someone to do something actually reduces the
sense of community, the valuing of good practice, that makes a healthy
society.

When I first applied the ccZero waiver to this blog I didnâ€™t really
think deeply about what I was doing. I wanted to make a point. I wanted
my work to be widely shared and I wanted to make it as easily shareable
as I could. In retrospect I can see I was making a statement about the
networked world I wanted to work in, one in which people actively
participate in building a better network. I was making the point that I
didnâ€™t just want to consume and benefit from the content, links, and
resources that other people had created, I wanted to give back. And I
have benefited, commercially, in the form of consultancies and grants,
and simply the opportunities that have opened up for me as a result of
reading and conversing about the work of other people.

My current life and work would be unthinkable without the network and
the value I have extracted from it. In return it is clear to me that I
need to give back in the form of resources that others are free to use,
and to exploit, even to make money off them. There may be a risk of
enclosure, although I think it small, but my choice as a citizen is to
be clear about what I expect of other citizens, not to attempt to
enforce my beliefs about good behaviour through legal documents but
through acting to build up and support the community of good citizens.

[Dave
White](http://www.conted.ox.ac.uk/staff/academicstaff/profile.php?a=alpha&id=8)
has
[talked](http://tallblog.conted.ox.ac.uk/index.php/2009/10/14/visitors-residents-the-video/)
and written about the [distinction between visitors and residents in
social
networks](http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/3171/3049),
the experience they bring and the experience they have. I think there is
a space, indeed a need, to recognize that there is another group beyond
those who simply inhabit online spaces. Those of us who want to build a
sustainable networked society should identify ourselves, our values, and
our expectations of others. Our networked world needs citizens as well

.

::: {.zemanta-pixie}
[![Enhanced by
Zemanta](http://img.zemanta.com/zemified_e.png?x-id=37245150-208f-4f49-90a8-b133c4245296){.zemanta-pixie-img
decoding="async"}](http://www.zemanta.com/ "Enhanced by Zemanta"){.zemanta-pixie-a}
:::


Image via Wikipedia  A few weeks ago I attended a workshop run by the ESRC Genomics Forum in Edinburgh which brought together humanists, social scientists, and science focused folks with an interest in how open approaches can and should be applied to genomic science. This was interesting on a number of levels but I was especially interested in the comments of Marina Levina on citizenship.

Messages de Rogue Scholar

Exploring HTTP Headers with netcat

Exploring HTTP Headers with netcat

A citizen of the network