Introduction
- The problem: some industries have poor worksite geolocation for structural reasons, e.g.:
- Telecom/utilities repair people: they often have a company car at their disposal and don't need to commute every day to the establishment they're registered to in RAIS
- Airline crew: all of them registered at the company's headquarters, even though many are based in other cities or even jumping from base to base.
- Therefore, including those sectors as-is introduces upward bias in accessibility measures.
- Current solution: sharp cutoff. We set the 95th percentile as the limit of a firm's number of employees in those industries
- Proposed solution: dynamic adjustment.
Motivation: number of firms and Zipf's law.
Tl;DR, we end up underestimating way too many firms at the end of the tail because of a handful big players, especially in sectors with large n (say, more than 1000 firms).
Example
| div_cnae20 |
p25 |
p50 |
p75 |
p95 |
p97 |
p99 |
p100 |
n_firms |
| 81 |
1 |
2 |
5 |
17 |
31 |
89 |
6397 |
7618 |
| 64 |
4 |
10 |
18 |
55 |
81 |
155 |
987 |
1016 |
| 42 |
2 |
8 |
26 |
169 |
253 |
663 |
4148 |
981 |
| 51 |
6 |
11 |
27 |
558 |
770 |
867 |
916 |
29 |
| 36 |
2 |
4 |
28 |
4608 |
7352 |
10096 |
11468 |
13 |
For instance, see the distribution of workers in sector 81 for the Belo Horizonte urban area in 2016: 99% of the firms have less than 90 employees, which seems reasonable and not that far from the rest of the distro.
But if we use the 95% sharp rule, we underestimate employment for firms in the (95,99) percentile interval because of the tiny bit in the (99,100] interval. Mathematically, we're replacing the 45298 workers in the (95,100] interval with $0.05 \cdot 7618 \cdot17 \approxeq 6475$ workers, which is only 14% of the original value.
Okay, so simply replace 95% with 99%, right?
No. Now look at div_cnae20 36: the big jump occurs between percentiles 75 and 95, hence, even the 95% cutoff is an overshooting. Therefore, we need to find programatically where the steepest changes are for each industry.
Methodology
First, three simple steps:
- Calculate the job distribution percentiles
- Divide them sequentially—e.g.,
p100/p99, p99/p95, p95/p75
- Choose the ratio with max value. If multiple ratios are equal, we keep the one that happens first (so
p95/p75 has preference over p99/p95)
And, finally, a fourth step which is the adjustment shenanigan.
Let $\ell_j$ be the number of workers of sector $j$'s $n$-th percentile, $r_j := t_j/\ell_j$ the max ratio described above, and $x_{ij}$ the number of workers at firm $i$, sector $j$. Then, if $x_{ij} > \ell_j$ , we correct the number of workers at firm with the rule
$$new_{ij} = \ell_j + \frac{x_{ij} - \ell_j}{r_j} .$$
Why the second term? Because when we divide $x$ by $new$, we get a concave, increasing"-ish" sequence that in the limit equals our initial ratio. In other words, we dilated a sudden jump in a larger interval, making it smoother.
More formally, if you substitute and simplify $x_{ir}/n_{ir}$ you'll end up with
$$f = \frac{ x_{ij} t_j }{ \ell_j t_j + \ell_j x_{ij} - \ell{j}^2},$$
and applying l'Hôpital's rule,
$$lim_{x \to \infty} f = lim_{x \to \infty} \frac{t}{\ell},$$
which is precisely our ratio $r$.
Application
It makes a lot more sense visually, so here is an example for four industries with different patterns. The new method isn't strictly above or below the sharp cutoff, but it produces smoother curves.
Introduction
Motivation: number of firms and Zipf's law.
Tl;DR, we end up underestimating way too many firms at the end of the tail because of a handful big players, especially in sectors with large
n(say, more than 1000 firms).Example
For instance, see the distribution of workers in sector 81 for the Belo Horizonte urban area in 2016: 99% of the firms have less than 90 employees, which seems reasonable and not that far from the rest of the distro.$0.05 \cdot 7618 \cdot17 \approxeq 6475$ workers, which is only 14% of the original value.
But if we use the 95% sharp rule, we underestimate employment for firms in the (95,99) percentile interval because of the tiny bit in the (99,100] interval. Mathematically, we're replacing the 45298 workers in the (95,100] interval with
Okay, so simply replace 95% with 99%, right?
No. Now look at
div_cnae2036: the big jump occurs between percentiles 75 and 95, hence, even the 95% cutoff is an overshooting. Therefore, we need to find programatically where the steepest changes are for each industry.Methodology
First, three simple steps:
p100/p99,p99/p95,p95/p75p95/p75has preference overp99/p95)And, finally, a fourth step which is the adjustment shenanigan.$\ell_j$ be the number of workers of sector $j$ 's $n$ -th percentile, $r_j := t_j/\ell_j$ the max ratio described above, and $x_{ij}$ the number of workers at firm $i$ , sector $j$ . Then, if $x_{ij} > \ell_j$ , we correct the number of workers at firm with the rule
Let
Why the second term? Because when we divide$x$ by $new$ , we get a concave, increasing"-ish" sequence that in the limit equals our initial ratio. In other words, we dilated a sudden jump in a larger interval, making it smoother.
More formally, if you substitute and simplify$x_{ir}/n_{ir}$ you'll end up with
and applying l'Hôpital's rule,
which is precisely our ratio$r$ .
Application
It makes a lot more sense visually, so here is an example for four industries with different patterns. The new method isn't strictly above or below the sharp cutoff, but it produces smoother curves.
