|
|
--- |
|
|
title: Data Collection China |
|
|
emoji: 🔥 |
|
|
colorFrom: blue |
|
|
colorTo: pink |
|
|
sdk: docker |
|
|
app_file: app.py |
|
|
pinned: false |
|
|
--- |
|
|
|
|
|
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference |
|
|
|
|
|
# Data Collection China |
|
|
|
|
|
This repository is dedicated to collecting and processing articles from various financial and governmental sources in China. It automates the scraping of data, summarizes content, vectorizes articles, and uploads them to a database for further analysis. |
|
|
|
|
|
## Project Structure |
|
|
|
|
|
- **/source**: Contains individual Python scripts for each data source. Each script has a `crawl` function to scrape articles from its respective source. |
|
|
- **/controllers**: |
|
|
- `summarizer.py`: Functions to summarize the content of articles. |
|
|
- `utils.py`: Utility functions to support data crawling and database operations. |
|
|
- `vectorizer.py`: Functions to vectorize articles and upload them to a vector database. |
|
|
- `main.py`: The main script that runs crawl functions from all source files. |
|
|
- `requirements.txt`: Lists all the dependencies necessary to run the scripts. |
|
|
|
|
|
## Data Sources |
|
|
|
|
|
The repository includes data collection scripts for the following sources: |
|
|
- **CSRC** (China Securities Regulatory Commission) |
|
|
- **CBIRC** (China Banking and Insurance Regulatory Commission) |
|
|
- **EastMoney** |
|
|
- **Gov** |
|
|
- **MOF** (Ministry of Finance) |
|
|
- **MOFCOM** (Ministry of Commerce) |
|
|
- **Stats** (National Bureau of Statistics of China) |
|
|
- **SAFE** (State Administration of Foreign Exchange) |
|
|
- **NDRC** (National Development and Reform Commission) |
|
|
|
|
|
## Installation |
|
|
|
|
|
1. Clone the repository: |
|
|
```bash |
|
|
git clone https://github.com/yourusername/data-collection-china.git |
|
|
2. Navigate to cloned directory |
|
|
```bash |
|
|
cd data-collection-china |
|
|
3. Install all dependencies |
|
|
```bash |
|
|
pip install -r requirements.txt |
|
|
4. Run main file |
|
|
```bash |
|
|
python main.py |
|
|
|
