Add latest stats and mca data#51
Conversation
tonyseek
left a comment
tonyseek
left a comment
There was a problem hiding this comment.
Thank you for your contribution.
Please squash your commits and refine the commit messages.
hugg95
changed the title
I have refined my commits |
|
Hi @hugg95, You could try to use The And then, I will checkout this pull request to local for checking the generated data. Thank you! |
Thanks, got it. |
1. add latest stats data 201810.tsv
2. add latest mca data 201904.tsv
3. update revisions.json and sources.tsv
mca/201904.tsv
revisions.json
sources.tsv
stats/201810.tsv
tonyseek
left a comment
tonyseek
left a comment
There was a problem hiding this comment.
Looks good for me. Thanks again for your contribution!
| mca 198212 http://files2.mca.gov.cn/www/201512/20151203093700735.htm mca | ||
| mca 198112 http://files2.mca.gov.cn/www/201512/20151203093558121.htm mca | ||
| mca 198012 http://files2.mca.gov.cn/www/201512/20151202154257477.htm mca | ||
| stats 201810 http://www.stats.gov.cn/tjsj/tjbz/tjyqhdmhcxhfdm/2018/index.html stats |
There was a problem hiding this comment.
It seems this URL could not be parsed by our script?
There was a problem hiding this comment.
It seems that the government no longer provide these stats data in old way and old format, I wrote a script myself crawling these stats data from this URL.
There was a problem hiding this comment.
We cannot access the URL "http://www.stats.gov.cn/tjsj/tjbz/xzqhdm/" now
There was a problem hiding this comment.
Hmm... The National Bureau of Statistics removed the legacy pages and provided a layered one, which is more detailed (night-digits) but not compatible with the old one. 😢
@lepture @timothyqiu, may we put the layered data into a new schema (e.g. stats-v2)?
@hugg95, would you merge your script into the scripts/parser.py?
There was a problem hiding this comment.
No problem, but I need time to update the script because its not designed for reuse...
There was a problem hiding this comment.
@tonyseek @lepture The nine-digit code has being around for quite a long time. As the name suggests, it's the division code for statistical use (统计用区划和城乡划分代码), which is different from administrative division code (行政区划代码). The Internet Archive page shows that they were two different standards listed under two different sections of the website.
So I believe it's not the National Bureau of Statistics replacing the legacy pages with more detailed ones. The Bureau is just quitting their role as a publisher of GB/T2260-compatible administrative division codes.
I think it's better to treat stats as a legacy data source which won't be updated any more. Then mca would be the only authoritative source of truth.
Anyway, GB/T2260 is, by definition, a six-digit code for divisions that are county level and above. Even if we decide to adopt the nine-digit source, divisions below county level should be dropped and the code should be truncated to conform to the standard.
Or maybe the nine-digit code should be a different feature / repository?
There was a problem hiding this comment.
@timothyqiu Got it. Then we just need to update the parser and archive old data (or use https://archive.org as well).
There was a problem hiding this comment.
@tonyseek May I submit a new parser script in this pull-request?
There was a problem hiding this comment.
@lepture @timothyqiu If there is no problem with the new data files, colud you please promote this pull request to be merged?
Thank you.
There was a problem hiding this comment.
@tonyseek @lepture @timothyqiu
Could you please help me to update to the latest GB2260 data? Now we have a web system that running on production environment using these data.
Thank you!
|
@hugg95 Could you send another PR for |
I created a new pull request for this script. Just for your information, thanks! |
4 participants
stats/201810.tsv，2018年统计用区划代码和城乡划分代码(截止2018年10月31日)mca/201904.tsv，2019年4月中华人民共和国县以上行政区划代码