I'm a backpacker sharing my journey into quantitative financial risks in the hopes that you don't have to suffer on your journey too.

Henrique Costa

<strong>
 Rambling (proxy for an introduction)
</strong>
We are experiencing the chaos of the information age, and its technological advents with voracious evolution and expansion. And in the midst of all this we have the advent of Big Data that has reached the delights of many with Data Science, Machine Learning and Artificial Intelligence. As a result, the idea of mathematical and statistical “models” became the rage, gaining visibility.

Trivia Series: Regression

Aqui voce pode encontrar a apresentação: Link da Apresentação     Aqui você se cadastra para participar do sorteio dos livros: Link para o Formulário                    You can also reproduce this by choosing the city you want. Check out the complete code on my Github.
<i>
</i>
Hey! 👋, did you find my work useful?

Sorte e Apostas

I started this website some time ago with the aim of writing down the expansion of my knowledge and experience in the area of financial risk management. And also other things like programming with R, econometric modeling and a little about
<em>
 data science
</em>
and
<em>
 machine learning
</em>
. So I’m going to use this publication as a starting point, and start discussing texts and examples applied to financial risks.

A few words about risk management

Having decided to improve my skills in the short space of time left between my professional life and personal life, I started researching the available data science options.

For me, the choice was R or Python

Na publicação de hoje, quero trazer uma brincadeira que fiz comigo mesmo, um pequeno incentivo para voltar a publicar conteúdos sobre R (eu já estava enferrujado) entre outros assunto sobre economia. Passeando pelo instagram, encontrei um perfil que pública muita informação sobre economia em formato de visualização, o Economista Visual.

Reproduzindo gráfico do Economista Visual - Valor da cesta básica (PT-BR)

In today’s post, I want to start a discussion about the use of mathematics and statistics in economics, and with this shocking sentence:  When a person studies economics for the first time, it is likely that they will not encounter “crazy” equations that go beyond basic mathematics.

The mathematization of Economics

When it comes to job titles, data scientist is one of the biggest buzzwords in recent years. It is also one of the fastest growing fields of professional activity. Can an economist really be a data scientist? What skills are needed?

Can Economists Become Data Scientists?

I will present here some personal reflections on a concept that is considered very simple and perhaps common in economics, but which is labeled as the height of ignorance for making an assumption that cannot be observed in reality, but which in my view is a mistake. mistake.

The Ceteris Paribus Condition

Risk Taker!

[![](https://hcostax.com/blog/regression/feature.png){.img-fluid}](feature.png){.lightbox
gallery="quarto-lightbox-gallery-1"}

::: {#rambling-proxy-for-an-introduction .section .level1}
# Rambling (proxy for an introduction)

We are experiencing the chaos of the information age, and its
technological advents with voracious evolution and expansion. And in the
midst of all this we have the advent of Big Data that has reached the
delights of many with Data Science, Machine Learning and Artificial
Intelligence.

As a result, the idea of mathematical and statistical "models" became
the rage, gaining visibility. And among these models we have the most
trivial of all: regression.

Have you ever stopped to think about where this term comes from, and
what it means in practice? Exactly what you thought: "That's the
basics!" That's why this series of stories was called "trivialities", as
they are trivialities that sometimes need to be known.

On my journey as a Quant backpacker (which is not that long, but long
enough to make these observations) I realized that very simple points or
concepts are not known to the overwhelming majority, and that is why I
observed a lot of misapplication or erroneous judgment in statistical
metrics , and I decided to start writing about it.

As I have academic roots, I brought the questions that will direct this
text:

- What is regression?
- What does this term mean?
- Where and when did it appear?
- How to apply?
- Am I applying it correctly?

And as usual, I'll start from the beginning (😄😄😄).
:::

::: {#the-beginnings-of-the-regression .section .level1}
# The beginnings of the Regression

When we think of this term, the idea of linear statistical regression
immediately comes to mind. And yes, that's right, the term is the same,
but why regression? When I think about the meaning of the word, and
obviously I already looked it up in a dictionary on the internet, which
is the same as going back, regression comes from regressing, returning
to a state that has already been overcome or that has evolved but
returned to its previous levels. previous ones. It's like climbing a
ladder but going down backwards.

And that's right (😄), regression is twisting, returning to the previous
state. This term was coined in 1875 by an amateur mathematician named
Francis Galton, who in some books says he was a first cousin of the
famous Charles Darwin. Galton brought to the public the coining of this
term, calling it "regression to the mean".

Why regression to the mean? Did something that moved end up returning to
the mean? Yes that's right. The concept only makes sense when we connect
it with classical statistics, or better, with the central limit theorem.

This theorem is a right arm of probability theory, and builds the
statement that when the sample size increases, the sampling distribution
gets closer and closer to a normal distribution, and this is fundamental
in statistical inference. Translating into simpler language: it's like
saying that when we observe nature, we perceive a natural state or a
standard, normal state of things.

An example using Brazil, a country with a tropical climate, it is
correct to say that the natural state of the climate is heat, as it is
sunny many more days a year than cold or rainy, therefore, on average it
is hotter, so the normal of Brazil is to have sunny days, but when it
rains we can activate the idea of normality, in which the rain can soon
stop and the sun and high temperatures return to the scene.

Regression to the mean is exactly that, even if you have days with rain,
the "normal" is that there are more sunny days, so rainy days are
deviations between the averages, as the average is sunny days. But the
term became known even after Galton published a study in 1885 in which
he demonstrated, through regression calculations, that the height of
children does not tend to reflect the height of their parents, but
rather tends to regress towards the population average.

It is possible to see how the term gained scale after use by Galton,
using the Google tool: Ngram book viewer.

[![](https://hcostax.com/blog/regression/img/ngram_regression.jpg){.img-fluid}](./img/ngram_regression.jpg){.lightbox
gallery="quarto-lightbox-gallery-2"}
:::

:::::::::::: {#the-use-of-regression .section .level1}
# The use of regression

The application of this method is known to many people, and can be found
in different professional areas. But as I come from economic sciences, I
will use this line to continue the text.

The economic sciences have econometrics as their quantitative tool,
which is the metric for analyzing economic theories. In simpler
language, it is the use of classical statistical techniques to analyze
and test an economic theory. Economists use a lot of regression in
econometrics.

And studying econometrics I learned a lot about the fundamentals of
regression analysis. But a few paragraphs above I wrote that regression
is returning to your previous state, so how would economists use a
matrix like that? For the simple fact that the economy works with the
assumption of a state of general equilibrium of things.

The economist has a single objective, and amazingly, it is not to save
money, it is to allocate scarce resources to meet unlimited needs. But
how does he do this? Artificial intelligence!!! (😄😄😄).

Joking apart. The economist uses optimization as his beacon. Allocating
scarce resources to satisfy unlimited needs is only possible by
optimizing. And what does optimizing have to do with regression? All!
Economists' assumption of a general equilibrium is only possible using
optimization, minimizing costs and maximizing profits, and regressing to
the mean or returning to a normal state only reinforces or supports the
idea of general equilibrium.

So, I will bring a brief example of application, coming from a study
material for economists, the book on basic econometrics by Damodar N.
Gujarati and Dawn C. Porter, in which we find an application that is
well known among economists: the hypothesis of marginal propensity to
consume, or MPC.

Keynes stated: ***The fundamental psychological law \[...\] is that men
\[women\] are disposed, as a rule and on an average, to increase their
consumption as their income increases, but not in the same proportion as
the increase in income.*** *(Keynes, John Maynard. The general theory of
employment, interest and money. New York: Harcourt Brace Jovanovich,
1936. p. 96.).*

So to simplify, we will use an econometric model based on a simple
linear regression, but first I need to comment that there is a rationale
behind a linear regression, which is the central objective of this text,
to talk about some basic points that sometimes fall into oblivion .

An econometric model (statistical/mathematical) aims to represent the
reality we are interested in investigating. The model must be able to
capture the relationships between reality and theory, so that the theory
can be tested. However, this representation of reality is not complete,
a model alone is not capable of representing or describing reality as a
whole, so it needs to be conditioned and sometimes restricted.

Keynes' hypothesis brings exactly this idea, with the construction of a
consumption function, and without many details, his hypothesis of a
relationship between consumption and income seems deterministic or an
exact relationship. So, as the model cannot have all the information, it
uses the most likely or available information and relies on the premise
that the other effects are constant or unchanged, not causing direct
influence: ceteris paribus.

I wrote a text about how I see the ceteris paribus condition, and how it
helps us understand real-life phenomena. So, returning to the example of
Keynes's hypothesis, he uses ceteris paribus in an intuitive and
implicit way, leaving no details, but establishing the premises: "to
increase your consumption as your income increases". So we already have
the relationship we need to build our model.

In short, Keynes postulated that the population had a tendency to
increase its consumption when its income increases, which was labeled
marginal propensity to consume (PMC), and it would be analyzed
quantitatively as a rate of variation in consumption, and that this
variation would be given in units (say, one dollar) of income, and that
it will always be greater than zero, but less than 1, as his
observations showed that additional consumption did not have the same
level as additional income, that is, people did not they consumed
everything they earned and that is why it has to be less than 1.

::::::::::: {#specifying-the-hypothesis-based-econometric-model .section .level2}
## Specifying the hypothesis-based econometric model {.anchored anchor-id="specifying-the-hypothesis-based-econometric-model"}

Although Keynes established an apparently positive relationship between
income and consumption, he did not stipulate how this relationship
happens. To simplify, let's use "poetic license" and suggest the
following functional form for the relationship between income and
consumption established by Keynes:

![](https://latex.codecogs.com/png.latex?%20Y%20=%20%5Cbeta_1%20+%20%5Cbeta_2X%20%5C;%5C;%5C;%5C;%5C;%5C;%5C;%5C;%200%20%3C%20%5Cbeta_2%20%3C%201%20)

where ![](https://latex.codecogs.com/png.latex?Y) represents consumption
expenditure and ![](https://latex.codecogs.com/png.latex?X) represents
income, and ![](https://latex.codecogs.com/png.latex?%5Cbeta_1) and
![](https://latex.codecogs.com/png.latex?%5Cbeta_2) are the model
parameters. These parameters represent the effects generated through the
relationship between ![](https://latex.codecogs.com/png.latex?X) and
![](https://latex.codecogs.com/png.latex?Y). Sometimes known as marginal
effects.

The parameter ![](https://latex.codecogs.com/png.latex?%5Cbeta_1) is
known as "intecept", or average value. Generally it demonstrates that
where the relationship between the variables between
![](https://latex.codecogs.com/png.latex?X) and
![](https://latex.codecogs.com/png.latex?Y) begins. It is the average
value, if the marginal effect is null (zero). It's like saying that, if
![](https://latex.codecogs.com/png.latex?X) has no influence on
![](https://latex.codecogs.com/png.latex?Y), then when
![](https://latex.codecogs.com/png.latex?X) and
![](https://latex.codecogs.com/png.latex?Y) are related, it is
![](https://latex.codecogs.com/png.latex?%5Cbeta_1) that shows the level
of this relationship.

The parameter ![](https://latex.codecogs.com/png.latex?%5Cbeta_2) is
known as "angle" or "angular coefficient", also called the slope of a
straight line, it determines the slope of a straight line. A straight
line because the relationship between
![](https://latex.codecogs.com/png.latex?X) and
![](https://latex.codecogs.com/png.latex?Y) is direct (linear). The
angular coefficient is a number that is related to the angle formed
between the straight line and the horizontal, describing the slope of
the straight line. And when we connect the terms "slope" and "angle", we
can remember the concept of derivation, and that is exactly it, it is
the effect of the derivation in relation to
![](https://latex.codecogs.com/png.latex?X). And we know that in
economics the concept of marginality is directly correlated to the
concept of derivation, which is why the angular coefficient is known as
marginal effect.

And the angular coefficient
(![](https://latex.codecogs.com/png.latex?%5Cbeta_2)) will be our
indicator of the relationship between income and consumption, which we
will call Marginal Propensity to Consume. This is already constructed by
Keynes' theory, but we are here trying to work on an idea.

As I mentioned in the paragraphs above, this relationship does not
represent a pure and exact reality. Of course, there are many variables
that can affect consumption in addition to income, and therefore the
model needs an additional specification, which for me is the charm of
the linear regression model.To take into account the influences of other
variables that were not imposed in this model:

![](https://latex.codecogs.com/png.latex?%20Y%20=%20%5Cbeta_1%20+%20%5Cbeta_2X%20+%20u%20)

where ![](https://latex.codecogs.com/png.latex?u) is known as the
disturbance, or error term of the model. It is a random (stochastic)
variable that has probabilistic properties. The error term
![](https://latex.codecogs.com/png.latex?u) is intended to represent all
other factors that affect consumption but are not explicitly imposed
here.

The error term is essentially the ceteris paribus condition. It contains
all the other effects that interact with Consumption, but they are being
nullified, or kept constant/unchanged, that is, there is no variation,
so no effect can be felt.

That is, by isolating the effect of income on consumption, we
artificially create a kind of general equilibrium, showing that the
relationship between ![](https://latex.codecogs.com/png.latex?X) and
![](https://latex.codecogs.com/png.latex?Y) is balanced (normal), and
that other factors have no influence.

With data from Table I.1 of the Gujarati basic econometrics book, which
refers to the United States economy in the period 1960-2005. Table I.1
uses aggregate consumption to represent model consumption, and GDP to
represent income, and thus we will estimate the marginal propensity to
consume. With the data in hand, it's time to go to RStudio and start
"the work". *Let's Code!*

::::: {.cell}
Show the code

::: {#cb1 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
# install.packages("janitor")
# remotes::install_github("brunoruas2/gujarati")

# load libraries
library(tidyverse)
library(gujarati)
# library(janitor)

TableI_1 %>% 
  mutate(
    Year = str_replace_all(Year, " ", ""),
    PCE.Y. = str_replace_all(PCE.Y., " ", ""),
    GDP.X. = str_replace_all(GDP.X., " ", ""),
  ) %>% 
  as_tibble(.name_repair = janitor::make_clean_names) %>% 
  mutate(
    pce_y = as.numeric(pce_y),
    gdp_x = as.numeric(gdp_x)
  ) -> tbl_pmc



model <- lm(pce_y ~ gdp_x, data = tbl_pmc)


model %>% 
  summary() %>% 
  pander::pander()
```
:::

::: {.cell-output-display}
|                 | Estimate | Std. Error | t value | Pr(\>\|t\|) |
|:---------------:|:--------:|:----------:|:-------:|:-----------:|
| **(Intercept)** |  -299.6  |   28.76    | -10.42  |  1.877e-13  |
|    **gdp_x**    |  0.7218  |  0.004423  |  163.2  |  7.244e-63  |

| Observations | Residual Std. Error | ![](https://latex.codecogs.com/png.latex?R%5E2) | Adjusted ![](https://latex.codecogs.com/png.latex?R%5E2) |
|:------------:|:-------------------:|:-----------------------------------------------:|:--------------------------------------------------------:|
|      46      |        73.57        |                     0.9984                      |                          0.9983                          |

Fitting linear model: pce_y \~ gdp_x
:::
:::::

Você também pode reproduzir isso escolhendo a cidade que quiser.

Confira o código completo no meu
[Github](https://github.com/hcostax/01---Desafio_EconomistaVisual).

::::::: {.callout .callout-style-default .callout-note .callout-titled}
::::: {.callout-header .d-flex .align-content-center}
::: {.callout-icon-container}
:::

::: {.callout-title-container .flex-fill}
Ei! 👋, você achou meu trabalho útil? Considere me comprar um café ☕,
clicando aqui 👇🏻
:::
:::::

::: {.callout-body-container .callout-body}
[![](https://img.buymeacoffee.com/button-api/?text=Buy%20me%20a%20coffee&emoji=&slug=hcostax&button_colour=5F7FFF&font_colour=ffffff&font_family=Cookie&outline_colour=000000&coffee_colour=FFDD00)](https://www.buymeacoffee.com/hcostax)
:::
:::::::
:::::::::::
::::::::::::

:::::::::: {#quarto-appendix .default}
::::: {#quarto-reuse .section .quarto-appendix-contents}
## Reuse {#reuse .anchored .quarto-appendix-heading}

:::: {.quarto-appendix-contents}
<div>

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/){rel="license"}

</div>
::::
:::::

:::::: {#quarto-citation .section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-costa2023 .csl-entry .quarto-appendix-citeas}
Costa, Henrique. 2023. "Trivia Series: Regression." December 20, 2023.
<https://doi.org/10.59350/xs70x-mar87>.
:::

</div>
::::::
::::::::::


[![](https://hcostax.com/blog/r_or_python/feature.jpeg){.img-fluid}](feature.jpeg){.lightbox
gallery="quarto-lightbox-gallery-1"}

Having decided to improve my skills in the short space of time left
between my professional life and personal life, I started researching
the available data science options. It quickly became clear that I
should choose a programming language to learn and apply the techniques
and the choice fell on R or Python, for me both options are good because
they are free and open source and have an excellent reputation, in
addition to each one having its own own community of active users, and
very active!

I don't remember where I saw this tip, but for me it was the central
factor in my decision and something worth sharing:

If you are a professional who uses analysis tools in a business context,
and with experience in Microsoft Excel, my tip is: select R to start
your journey in data science. R is a single-threaded object-oriented
functional programming language, and once you understand the main
commands, it is intuitive to use. As such, it's pretty predictable,
which has been great for me as a professional. R has an excellent
graphics package that complements the idea of analyzing data or *data
wrangling*, which Python programmers often turn to *ggplot2* to generate
beautiful visualizations, or use the ggplot2 theme within Python (lol),
even though in recent years the Python's visualization libraries have
improved a lot, yet this happens frequently.

On the other hand, if you already have some experience in programming,
or are from the computing field, Python is a much more general purpose
and more readable language for those coming from this field, and
everything you can do in Python is the same as most of the things R is
good at.

Both groups have strong communities that share their knowledge on
various blogs and events and I really think they are both great choices.
Today the market is more heated when it comes to Python, and
professionals who know both languages do very well.

Now that I've been using R for a few years (since 2013), I've had the
chance to use it a few times in the workplace, and I use it whenever I
can, showing the resources this tool offers, creating econometric models
and estimation procedures. in modeling for financial risks to provide
some business *insights*. I have already carried out many analysis
projects for private clients and I feel honored that my efforts to learn
are generating results.

A fellow Python expert with an interest in machine learning once saw the
few lines of code needed to organize the dataset, train a model, and
predict results, and, frankly, he was shocked. We had an awkward
R-to-Python conversation trying to understand the differences between
data matrices and data frames afterwards, although for me it clarified
where R was strong: the ease of preparing and building a model quickly
for any type of analysis.

But depending on who you ask, when I first came into contact with R in
2013, R probably had a slight, if not substantial, advantage over Python
in user adoption for machine learning and what is now known as data
science. Since then, the use of Python has grown substantially and it
would be difficult to argue against the fact that Python is the new
favorite, although the race may be tighter than one might expect given
the enthusiasm of Python fans supporting the new and Brilliant tool with
biggest hype.

In recent years, Python has benefited greatly from the rapid maturation
of free add-ons such as the *Scikit-learn* machine learning framework,
the *Pandas* data structure library, the *Matplotlib* graphing library,
and the *Jupyter* notebook interface, among several other open
applications. Source libraries that make it easier than ever to do data
science in Python. Of course, these libraries only brought Python to par
with what R and RStudio could do long ago! However, Python is
comparatively fast and memory efficient -- at least relative to R --
which may have contributed to the fact that Python is now arguably the
language most frequently taught in formal programs in data science. and
quickly gained adoption across business domains.

Rather than indicating the imminent death of R, because Python's rise is
steep. In fact, the use of R is also growing rapidly, and R and RStudio
are becoming more popular than ever. Although students sometimes ask
whether it's worth starting with R rather than jumping straight to
Python, there are still many good reasons to choose to learn machine
learning with R over the alternative.

Please note that these justifications are quite subjective -- not just
mine, but any justification on the internet will be like this -- and
there is no right answer for everyone, so I hesitate to put this in
writing! However, as someone who still uses R almost daily as part of my
work for a large corporation, here are a few things I've noticed:

- As I mentioned above, R may be more intuitive and easier to learn for
  people with a background in social sciences or business (such as
  economics, marketing, and so on), while Python may make more sense for
  computer scientists and other types of engineers .

- R tends to be used more like a "calculator" where you type a command
  and something happens; In general, coding in Python tends to require
  more consideration of loops and other program flow commands (this
  distinction is disappearing over time with additional functionality in
  Python libraries).

- R uses relatively few types of data structures (those included are
  adapted for data analysis) and the frequently used spreadsheet-type
  data format is a built-in data type; Comparatively, Python has many
  specialized data structures and uses libraries like *NumPy* or
  *Pandas* , and for array data format, each has its own syntax.

- R and its packages can be easier to install and update than Python, in
  part because Python is managed by some operating systems by default,
  and maintaining separate dependencies and environments is a challenge
  (modern Python installation tools and managers of packages addressed
  this simultaneously, in some ways making the problem worse!).

- R is typically slower and more memory-intensive than Python for data
  manipulation and iteration over large data structures, but if the data
  fits in memory, this difference is somewhat insignificant. For real?
  If you don't work with BigData, you probably don't even feel this
  difference; R has improved in this area, R is making data preparation
  faster and easier, and for data that doesn't fit in memory there are
  workarounds (R with BigData), but this is admittedly one of Python's
  biggest advantages.

- R has the support and vision of the Posit team (formerly known as
  RStudio) driving innovation and making R easier and more enjoyable to
  use in a unified software environment (RStudio Desktop); In contrast,
  Python's innovations are occurring on multiple fronts, offering more
  "right" ways to accomplish the same thing (for better or worse).

Hopefully, the reasons above will give you the confidence to begin your
journey. There is no shame in starting with R (as some people think),
whether you stay with R long-term, use it side-by-side with other
languages like Python, or major in something entirely different, the
fundamental principles you learn will be transferred to any language (a
clear example is: if you haven't yet learned SQL and already understand
the basics of *Tidyverse*, you will have an easier time studying SQL, or
vice versa) or tools you choose. Although code written in R is much
"prettier" to me, I highly recommend that you use the right tool for the
job, whatever it may be. You may find, as I did, that R and RStudio are
your tools of choice for many real-world data science and machine
learning projects - even if you occasionally take advantage of Python's
unique strengths!

In the end, my recommendation is: just pick one and get started.
Although I will still need to learn a lot more about Python to get the
unique benefits each language offers, amplified through collaboration.
The good thing about this is that, using RStudio, an IDE for R, it is
possible to use Python and SQL as well.

::::::: {.callout .callout-style-default .callout-note .callout-titled}
::::: {.callout-header .d-flex .align-content-center}
::: {.callout-icon-container}
:::

::: {.callout-title-container .flex-fill}
Hey! 👋, did you find my work useful? Consider buying me a coffee ☕, by
clicking here 👇🏻
:::
:::::

::: {.callout-body-container .callout-body}
[![](https://img.buymeacoffee.com/button-api/?text=Buy%20me%20a%20coffee&emoji=&slug%20=hcostax&button_colour=5F7FFF&font_colour=ffffff&font_family=Cookie&outline_colour=000000&coffee_colour=FFDD00)](https://www.buymeacoffee.com/hcostax)
:::
:::::::

:::::::::: {#quarto-appendix .default}
::::: {#quarto-reuse .section .quarto-appendix-contents}
## Reuse {#reuse .anchored .quarto-appendix-heading}

:::: {.quarto-appendix-contents}
<div>

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/){rel="license"}

</div>
::::
:::::

:::::: {#quarto-citation .section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-costa2023 .csl-entry .quarto-appendix-citeas}
Costa, Henrique. 2023. "For Me, the Choice Was R or Python." June 20,
2023. <https://doi.org/10.59350/83vew-1de35>.
:::

</div>
::::::
::::::::::
