fix NotValidLengthException

Add Data Source [CCEF]

Add Data Source [CCEF]

refactor logging and streamline content update process

creation of flask api

fix(eastmoney)

Return empty string on translation failure in translate function

Update utils.py

Increase timeout for URL requests in crawl functions to enhance reliability

Add timeout parameter to URL requests in crawl functions for improved reliability

Handle LangDetectException in crawl_by_url function to improve error handling

Refactor vectorization process by removing openai_vectorize calls and updating vectorizer initialization

Add validation for content length and enhance error handling in crawl_by_url function

Merge branch 'main' of https://github.com/oxbridge-econ/data-collection-china

Add handling for DependencyError in PDF extraction and update requirements to include pycryptodome

Update utils.py

Refactor content update process to ensure reference ID is set to None and re-enable vectorization functions in article processing

Fix table name casing in update_content function for DynamoDB

Add reference ID extraction and implement retry logic for document addition

Increase retry attempts and adjust sleep duration for translation requests

Refactor translation error handling and remove debug print statements in vectorization

Implement retry logic for translation requests to handle RequestError exceptions

Replace logging with print statements for content update and reference extraction functions

Add logging configuration and info statements for content updates and reference extraction

Limit content length to 500 characters in sentiment computation for improved analysis accuracy

Refactor exception handling in multiple files to specify exception types and improve logging

Refactor error handling and improve logging in utils.py; update vectorization process in vectorizer.py; adjust variable naming in eastmoney.py

commit

commit

commit

commit

commit

commit

commit

commit