metadata

title: Data Collection China
emoji: 🔥
colorFrom: blue
colorTo: pink
sdk: docker
app_file: app.py
pinned: false

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

Data Collection China

This repository is dedicated to collecting and processing articles from various financial and governmental sources in China. It automates the scraping of data, summarizes content, vectorizes articles, and uploads them to a database for further analysis.

Project Structure

/source: Contains individual Python scripts for each data source. Each script has a crawl function to scrape articles from its respective source.
/controllers:
- summarizer.py: Functions to summarize the content of articles.
- utils.py: Utility functions to support data crawling and database operations.
- vectorizer.py: Functions to vectorize articles and upload them to a vector database.
main.py: The main script that runs crawl functions from all source files.
requirements.txt: Lists all the dependencies necessary to run the scripts.

Data Sources

The repository includes data collection scripts for the following sources:

CSRC (China Securities Regulatory Commission)
CBIRC (China Banking and Insurance Regulatory Commission)
EastMoney
Gov
MOF (Ministry of Finance)
MOFCOM (Ministry of Commerce)
Stats (National Bureau of Statistics of China)
SAFE (State Administration of Foreign Exchange)
NDRC (National Development and Reform Commission)

Installation

Clone the repository:

git clone https://github.com/yourusername/data-collection-china.git

Navigate to cloned directory
```
cd data-collection-china
```
Install all dependencies
```
pip install -r requirements.txt
```
Run main file
```
python main.py
```