Initial commit for Hugging Face Spaces

.gitignore

@@ -0,0 +1,6 @@
+.venv/
+# add pycache and pyc files
+__pycache__/
+*.pyc
+# add lib directory
+lib/

.python-version

@@ -0,0 +1 @@
+3.11

README.md

@@ -0,0 +1,61 @@
+---
+title: >-
+  Unveiling Global Narratives Through Knowledge Graphs: A Case Study Using GDELT
+  and Streamlit
+emoji: 🔮
+colorFrom: indigo
+colorTo: blue
+sdk: streamlit
+sdk_version: 1.42.0
+app_file: app.py
+pinned: false
+license: cc-by-4.0
+short_description: using knowledge graphs for insight
+---
+
+**Title:** Unveiling Global Narratives Through Knowledge Graphs: A Case Study Using GDELT and Streamlit
+**Keywords:** GDELT, Knowledge Graphs, Network Analysis, Sentiment Analysis, Prefect, Hugging Face datasets, DuckDB, Streamlit, Neo4j, NetworkX, st-link-analysis, streamlit-aggrid, pyvis, pandas
+
+## Abstract  
+The global landscape is increasingly shaped by evolving narratives driven by interconnected events and entities. To better understand these dynamics, we introduce **GDELT Insight Explorer**, a knowledge graph-based platform built using Streamlit, DuckDB, and NetworkX. This paper presents a detailed case study on using the platform to analyze GDELT Global Knowledge Graph (GKG) data from March 2020. We focus on uncovering global narratives and relationships between actors and themes during the early phase of the COVID-19 pandemic. Our findings emphasize the utility of real-time event data visualization and network analysis in tracing narrative propagation and identifying key influencers in global events.
+
+## 1. Introduction  
+Understanding global narratives requires tools that can capture the complexity of events, their associated entities, and evolving sentiment over time. Traditional tabular analysis methods are often insufficient for capturing these relationships at scale. Knowledge graphs offer a robust solution for modeling and visualizing the interconnected nature of real-world events. This paper documents the development and application of **GDELT Insight Explorer**, a platform designed to leverage GDELT data for interactive exploration and insight generation.
+
+## 2. Methodology  
+
+### 2.1 Data Source and Processing  
+The application is powered by the GDELT Global Knowledge Graph (GKG) dataset, focusing on data from March 10–22, 2020. The dataset includes key features such as themes, locations, persons, organizations, and sentiment scores. Our ETL pipeline, implemented using Prefect and DuckDB, extracts and transforms the data into a Parquet format for efficient querying and filtering.
+
+- **Data Filtering:** We prioritize events with a tone score below -6 to identify highly negative narratives.
+- **Data Storage:** DuckDB is used for in-memory querying, enabling real-time analysis of filtered datasets.
+- **Graph Construction:** NetworkX and Neo4j are employed for graph creation, with relationships categorized into entities such as persons, organizations, and locations.
+
+### 2.2 Platform Architecture  
+The platform is built using Streamlit, with a modular architecture that supports multiple analysis modes:
+- **Event Navigator:** Provides a tabular overview of filtered events with interactive search and filtering.
+- **Event Graph Explorer:** Visualizes events and their associated entities in a graph format.
+- **Community Detection and Network Analysis:** Employs NetworkX to detect communities and analyze network metrics such as centrality and density.
+
+## 3. Findings  
+
+### 3.1 Narrative Detection and Sentiment Analysis  
+The negative tone filter helped identify early COVID-related narratives, revealing clusters of related events involving key global actors. By visualizing these relationships, we observed recurring themes of public health concerns and geopolitical tensions.
+
+### 3.2 Community Detection  
+Using the Louvain method for community detection, we identified cohesive subgroups within the network. These communities often corresponded to specific geographic regions or thematic clusters, providing deeper insights into localized narratives.
+
+### 3.3 Real-Time Filtering and Exploration  
+The integration of DuckDB allowed for seamless data filtering and exploration within the Streamlit interface. Users could drill down from high-level overviews to individual event records, facilitating rapid insight generation.
+
+## 4. Conclusion and Future Work  
+The **GDELT Insight Explorer** demonstrates the potential of combining knowledge graphs and real-time data exploration for uncovering global narratives. Future work will focus on expanding the temporal range of the dataset, integrating additional data sources, and incorporating machine learning models for predictive analysis. The open-source nature of the platform encourages further development and adaptation across different domains.
+
+## References  
+1. GDELT Project. (n.d.). [https://www.gdeltproject.org](https://www.gdeltproject.org)  
+2. Newman, M. E. J. (2018). *Networks: An Introduction*. Oxford University Press.  
+3. DuckDB. (n.d.). [https://duckdb.org](https://duckdb.org)  
+4. Prefect. (n.d.). [https://www.prefect.io](https://www.prefect.io)
+
+## Appendix: Application Architecture and Code  
+For implementation details, please refer to the open-source repository: [https://huggingface.co/spaces/dwb2023/insight](https://huggingface.co/spaces/dwb2023/insight).

app.py

@@ -0,0 +1,24 @@
+import streamlit as st
+
+st.set_page_config(
+    page_title="GDELT Insight Explorer",
+    layout="wide",
+    page_icon="🔮"
+)
+
+st.title("GDELT Insight Explorer: Unveiling Global Event Narratives")
+st.markdown("""
+Welcome to the **GDELT Insight Explorer**, a multi-faceted platform that leverages knowledge graph techniques to analyze global events and trends.
+
+**How to Get Started:**
+- Use the sidebar to switch between different analysis modes.
+- Explore datasets, visualize event relationships, and analyze network structures.
+
+**Available Pages:**
+- **🗺️ COVID Navigator:** Dive into curated COVID-related event data.
+- **🔍 COVID Event Graph Explorer:** Inspect detailed event records and their interconnections.
+- **🌐 Global Network Analysis:** Visualize and analyze the global network of events.
+- **🗺️ Feb 2025 Navigator:** Investigate recent event data with advanced filtering.
+- **🔍 Feb 2025 Event Graph Explorer:** Inspect detailed event records and their interconnections.            
+- **🧪 Feb 2025 Dataset Experimentation:** An experiment using the HF dataset directly to investigate impact on query behavior and performance.
+""")

data_access.py

@@ -0,0 +1,118 @@
+"""
+Data access module for GDELT data retrieval and filtering
+"""
+import duckdb
+import pandas as pd
+
+def get_gdelt_data(
+    limit=10,
+    tone_threshold=-7.0,
+    start_date=None,
+    end_date=None,
+    source_filter=None,
+    themes_filter=None,
+    persons_filter=None,
+    organizations_filter=None,
+    locations_filter=None
+):
+    """Get filtered GDELT data from DuckDB with dynamic query parameters."""
+    con = duckdb.connect(database=':memory:')
+
+    # Create view of the dataset
+    con.execute("""
+        CREATE VIEW negative_tone AS (
+            SELECT * 
+            FROM read_parquet('hf://datasets/dwb2023/gdelt-gkg-march2020-v2@~parquet/default/negative_tone/*.parquet')
+        );
+    """)
+
+    # Base query components
+    base_conditions = [
+        "SourceCollectionIdentifier IS NOT NULL",
+        "DATE IS NOT NULL",
+        "SourceCommonName IS NOT NULL",
+        "DocumentIdentifier IS NOT NULL",
+        "V1Counts IS NOT NULL",
+        "V1Themes IS NOT NULL",
+        "V1Locations IS NOT NULL",
+        "V1Persons IS NOT NULL",
+        "V1Organizations IS NOT NULL",
+        "V2GCAM IS NOT NULL",
+        "\"V2.1Quotations\" IS NOT NULL",
+        "tone <= ?"
+    ]
+    params = [tone_threshold]
+    extra_conditions = []
+
+    # Add optional filters
+    if start_date:
+        extra_conditions.append("DATE >= ?")
+        params.append(start_date)
+    if end_date:
+        extra_conditions.append("DATE <= ?")
+        params.append(end_date)
+    if source_filter:
+        extra_conditions.append("SourceCommonName ILIKE ?")
+        params.append(f"%{source_filter}%")
+    if themes_filter:
+        extra_conditions.append("(V1Themes ILIKE ? OR V2EnhancedThemes ILIKE ?)")
+        params.extend([f"%{themes_filter}%", f"%{themes_filter}%"])
+    if persons_filter:
+        extra_conditions.append("(V1Persons ILIKE ? OR V2EnhancedPersons ILIKE ?)")
+        params.extend([f"%{persons_filter}%", f"%{persons_filter}%"])
+    if organizations_filter:
+        extra_conditions.append("(V1Organizations ILIKE ? OR V2EnhancedOrganizations ILIKE ?)")
+        params.extend([f"%{organizations_filter}%", f"%{organizations_filter}%"])
+    if locations_filter:
+        extra_conditions.append("(V1Locations ILIKE ? OR V2EnhancedLocations ILIKE ?)")
+        params.extend([f"%{locations_filter}%", f"%{locations_filter}%"])
+
+    # Combine all conditions
+    all_conditions = base_conditions + extra_conditions
+    where_clause = " AND ".join(all_conditions) if all_conditions else "1=1"
+
+    # Build final query
+    query = f"""
+        SELECT *
+        FROM negative_tone
+        WHERE {where_clause}
+        LIMIT ?;
+    """
+    params.append(limit)
+
+    # Execute query with parameters
+    results_df = con.execute(query, params).fetchdf()
+    con.close()
+    
+    return results_df
+
+def filter_dataframe(df, source_filter=None, date_filter=None, tone_min=None, tone_max=None):
+    """Filter dataframe based on provided criteria"""
+    display_df = df[['GKGRECORDID', 'DATE', 'SourceCommonName', 'tone']].copy()
+    display_df.columns = ['ID', 'Date', 'Source', 'Tone']
+
+    if source_filter:
+        display_df = display_df[display_df['Source'].str.contains(source_filter, case=False, na=False)]
+    if date_filter:
+        display_df = display_df[display_df['Date'].str.contains(date_filter, na=False)]
+    if tone_min is not None and tone_max is not None:
+        display_df = display_df[
+            (display_df['Tone'] >= tone_min) & 
+            (display_df['Tone'] <= tone_max)
+        ]
+
+    return display_df
+
+# Constants for raw data categories
+GDELT_CATEGORIES = {
+    "Metadata": ["GKGRECORDID", "DATE", "SourceCommonName", "DocumentIdentifier", "V2.1Quotations", "tone"],
+    "Persons": ["V2EnhancedPersons", "V1Persons"],
+    "Organizations": ["V2EnhancedOrganizations", "V1Organizations"],
+    "Locations": ["V2EnhancedLocations", "V1Locations"],
+    "Themes": ["V2EnhancedThemes", "V1Themes"],
+    "Names": ["V2.1AllNames"],
+    "Counts": ["V2.1Counts", "V1Counts"],
+    "Amounts": ["V2.1Amounts"],
+    "V2GCAM": ["V2GCAM"],
+    "V2.1EnhancedDates": ["V2.1EnhancedDates"],
+}

graph_builder.py

@@ -0,0 +1,220 @@
+"""
+Graph builder module for converting GDELT data to graph formats
+"""
+import pandas as pd
+import networkx as nx
+import json
+
+class GraphBuilder:
+    """Base class for building graph from GDELT data"""
+    def process_entities(self, row):
+        """Process entities from a row and return nodes and relationships"""
+        nodes = []
+        relationships = []
+        event_id = row["GKGRECORDID"]
+        event_date = row["DATE"]
+        event_source = row["SourceCommonName"]
+        event_document_id = row["DocumentIdentifier"]
+        # event_image = row["V2.1SharingImage"] if pd.notna(row["V2.1SharingImage"]) else ""
+        event_quotations = row["V2.1Quotations"] if pd.notna(row["V2.1Quotations"]) else ""
+        event_tone = float(row["tone"]) if pd.notna(row["tone"]) else 0.0
+
+        # Add event node
+        nodes.append({
+            "id": event_id,
+            "type": "event",
+            "properties": {
+                "date": event_date,
+                "source": event_source,
+                "document": event_document_id,
+                # "image": event_image,
+                "quotations": event_quotations,
+                "tone": event_tone
+            }
+        })
+
+        # Process each entity type
+        entity_mappings = {
+            "V2EnhancedPersons": ("Person", "MENTIONED_IN"),
+            "V2EnhancedOrganizations": ("Organization", "MENTIONED_IN"),
+            "V2EnhancedLocations": ("Location", "LOCATED_IN"),
+            "V2EnhancedThemes": ("Theme", "CATEGORIZED_AS"),
+            "V2.1AllNames": ("Name", "MENTIONED_IN"),
+            "V2.1Counts": ("Count", "MENTIONED_IN"),
+            "V2.1Amounts": ("Amount", "MENTIONED_IN"),
+        }
+
+        for field, (label, relationship) in entity_mappings.items():
+            if pd.notna(row[field]):
+                entities = [e.strip() for e in row[field].split(';') if e.strip()]
+                for entity in entities:
+                    nodes.append({
+                        "id": entity,
+                        "type": label.lower(),
+                        "properties": {"name": entity}
+                    })
+                    relationships.append({
+                        "from": entity,
+                        "to": event_id,
+                        "type": relationship,
+                        "properties": {"created_at": event_date}
+                    })
+
+        return nodes, relationships
+
+class NetworkXBuilder(GraphBuilder):
+    """Builder for NetworkX graphs"""
+    def build_graph(self, df):
+        G = nx.Graph()
+
+        for _, row in df.iterrows():
+            nodes, relationships = self.process_entities(row)
+
+            # Add nodes
+            for node in nodes:
+                G.add_node(node["id"],
+                          type=node["type"],
+                          **node["properties"])
+
+            # Add relationships
+            for rel in relationships:
+                G.add_edge(rel["from"],
+                          rel["to"],
+                          relationship=rel["type"],
+                          **rel["properties"])
+
+        return G
+
+class Neo4jBuilder(GraphBuilder):
+    def __init__(self, uri, user, password):
+        self.driver = GraphDatabase.driver(uri, auth=(user, password))
+        self.logger = logging.getLogger(__name__)
+
+    def close(self):
+        self.driver.close()
+
+    def build_graph(self, df):
+        with self.driver.session() as session:
+            for _, row in df.iterrows():
+                nodes, relationships = self.process_entities(row)
+
+                # Create nodes and relationships in Neo4j
+                try:
+                    session.execute_write(self._create_graph_elements,
+                                        nodes, relationships)
+                except Exception as e:
+                    self.logger.error(f"Error processing row {row['GKGRECORDID']}: {str(e)}")
+
+    def _create_graph_elements(self, tx, nodes, relationships):
+        # Create nodes
+        for node in nodes:
+            query = f"""
+            MERGE (n:{node['type']} {{id: $id}})
+            SET n += $properties
+            """
+            tx.run(query, id=node["id"], properties=node["properties"])
+
+        # Create relationships
+        for rel in relationships:
+            query = f"""
+            MATCH (a {{id: $from_id}})
+            MATCH (b {{id: $to_id}})
+            MERGE (a)-[r:{rel['type']}]->(b)
+            SET r += $properties
+            """
+            tx.run(query,
+                  from_id=rel["from"],
+                  to_id=rel["to"],
+                  properties=rel["properties"])
+
+class StreamlitGraphBuilder:
+    """Adapted graph builder for Streamlit visualization"""
+    def __init__(self):
+        self.G = nx.Graph()
+        
+    def process_row(self, row):
+        """Process a single row of data"""
+        event_id = row["GKGRECORDID"]
+        event_props = {
+            "type": "event",  # already in lowercase
+            "date": row["DATE"],
+            "source": row["SourceCommonName"],
+            "document": row["DocumentIdentifier"],
+            "tone": row["tone"],
+            # Store display name in its original format if needed.
+            "name": row["SourceCommonName"]
+        }
+        
+        self.G.add_node(event_id, **event_props)
+        
+        # Use lowercase node types for consistency in lookups.
+        entity_types = {
+            "V2EnhancedPersons": ("person", "MENTIONED_IN"),
+            "V2EnhancedOrganizations": ("organization", "MENTIONED_IN"),
+            "V2EnhancedLocations": ("location", "LOCATED_IN"),
+            "V2EnhancedThemes": ("theme", "CATEGORIZED_AS"),
+            "V2.1AllNames": ("name", "MENTIONED_IN"),
+            "V2.1Counts": ("count", "MENTIONED_IN"),
+            "V2.1Amounts": ("amount", "MENTIONED_IN"),
+        }
+        
+        for col, (node_type, rel_type) in entity_types.items():
+            if pd.notna(row[col]):
+                # The actual display value (which may be in Parent Case) is preserved in the "name" attribute.
+                entities = [e.strip() for e in row[col].split(';') if e.strip()]
+                for entity in entities:
+                    self.G.add_node(entity, type=node_type, name=entity)
+                    self.G.add_edge(entity, event_id, 
+                                    relationship=rel_type, 
+                                    date=row["DATE"])
+
+class StLinkBuilder(GraphBuilder):
+    """Builder for st-link-analysis compatible graphs"""
+    def build_graph(self, df):
+        """Build graph in st-link-analysis format"""
+        all_nodes = []
+        all_edges = []
+        edge_counter = 0
+
+        # Track nodes we've already added to avoid duplicates
+        added_nodes = set()
+        
+        for _, row in df.iterrows():
+            nodes, relationships = self.process_entities(row)
+            
+            # Process nodes
+            for node in nodes:
+                if node["id"] not in added_nodes:
+                    stlink_node = {
+                        "data": {
+                            "id": str(node["id"]),
+                            "label": node["type"].upper(),
+                            **node["properties"]
+                        }
+                    }
+                    all_nodes.append(stlink_node)
+                    added_nodes.add(node["id"])
+            
+            # Process relationships/edges
+            for rel in relationships:
+                edge_counter += 1
+                stlink_edge = {
+                    "data": {
+                        "id": f"e{edge_counter}",
+                        "source": str(rel["from"]),
+                        "target": str(rel["to"]),
+                        "label": rel["type"],
+                        **rel["properties"]
+                    }
+                }
+                all_edges.append(stlink_edge)
+        
+        return {
+            "nodes": all_nodes,
+            "edges": all_edges
+        }
+
+    def write_json(self, graph_data, filename):
+        """Write graph to JSON file"""
+        with open(filename, 'w') as f:
+            json.dump(graph_data, f, indent=2)

graph_config.py

@@ -0,0 +1,60 @@
+"""
+Configuration module for graph visualization styles
+"""
+from st_link_analysis import NodeStyle, EdgeStyle
+
+# Node styles configuration
+NODE_STYLES = [
+    NodeStyle("EVENT", "#FF7F3E", "name", "description"),
+    NodeStyle("PERSON", "#4CAF50", "name", "person"),
+    NodeStyle("ORGANIZATION", "#9C27B0", "name", "business"),
+    NodeStyle("LOCATION", "#2196F3", "name", "place"),
+    NodeStyle("THEME", "#FFC107", "name", "sell"),
+    NodeStyle("COUNT", "#795548", "name", "inventory"),
+    NodeStyle("AMOUNT", "#607D8B", "name", "wallet"),
+]
+
+NODE_TYPES = {
+    'event': {
+        'color': '#1f77b4',
+        'description': 'GDELT Events'
+    },
+    'person': {
+        'color': '#2ca02c',
+        'description': 'Named Persons'
+    },
+    'organization': {
+        'color': '#ffa500',
+        'description': 'Organizations'
+    },
+    'location': {
+        'color': '#ff0000',
+        'description': 'Geographic Locations'
+    },
+    'theme': {
+        'color': '#800080',
+        'description': 'Event Themes'
+    }
+}
+
+# Edge styles configuration
+EDGE_STYLES = [
+    EdgeStyle("MENTIONED_IN", caption="label", directed=True),
+    EdgeStyle("LOCATED_IN", caption="label", directed=True),
+    EdgeStyle("CATEGORIZED_AS", caption="label", directed=True)
+]
+
+# Layout options
+LAYOUT_OPTIONS = ["cose", "circle", "grid", "breadthfirst", "concentric"]
+
+# Default graph display settings
+DEFAULT_GRAPH_HEIGHT = 500
+DEFAULT_LAYOUT = "cose"
+
+# Column configuration for data grid
+GRID_COLUMNS = {
+    "ID": {"width": "medium"},
+    "Date": {"width": "small"},
+    "Source": {"width": "medium"},
+    "Tone": {"width": "small", "format": "%.2f"}
+}

pages/1_🗺️_COVID_Navigator.py

@@ -0,0 +1,174 @@
+import streamlit as st
+import duckdb
+import pandas as pd
+from st_aggrid import AgGrid, GridOptionsBuilder, GridUpdateMode
+
+# Constants for raw data categories
+GDELT_CATEGORIES = {
+    "Metadata": ["GKGRECORDID", "DATE", "SourceCommonName", "DocumentIdentifier", "V2.1Quotations", "tone"],
+    "Persons": ["V2EnhancedPersons", "V1Persons"],
+    "Organizations": ["V2EnhancedOrganizations", "V1Organizations"],
+    "Locations": ["V2EnhancedLocations", "V1Locations"],
+    "Themes": ["V2EnhancedThemes", "V1Themes"],
+    "Names": ["V2.1AllNames"],
+    "Counts": ["V2.1Counts", "V1Counts"],
+    "Amounts": ["V2.1Amounts"],
+    "V2GCAM": ["V2GCAM"],
+    "V2.1EnhancedDates": ["V2.1EnhancedDates"],
+}
+
+def initialize_db():
+    """Initialize database connection and create dataset view"""
+    con = duckdb.connect()
+    con.execute("""
+        CREATE VIEW negative_tone AS (
+            SELECT * 
+            FROM read_parquet('hf://datasets/dwb2023/gdelt-gkg-march2020-v2@~parquet/default/negative_tone/*.parquet')
+        );
+    """)
+    return con
+
+def fetch_data(con, source_filter=None, themes_filter=None, 
+               start_date=None, end_date=None, limit=50, include_all_columns=False):
+    """Fetch filtered data from the database"""
+    if include_all_columns:
+        columns = "*"
+    else:
+        columns = "GKGRECORDID, DATE, SourceCommonName, tone, DocumentIdentifier, 'V2.1Quotations', SourceCollectionIdentifier"
+    
+    query = f"""
+        SELECT {columns}
+        FROM negative_tone 
+        WHERE TRUE
+    """
+    params = []
+    
+    if source_filter:
+        query += " AND SourceCommonName ILIKE ?"
+        params.append(f"%{source_filter}%")
+    if start_date:
+        query += " AND DATE >= ?"
+        params.append(start_date)
+    if end_date:
+        query += " AND DATE <= ?"
+        params.append(end_date)
+    if limit:
+        query += f" LIMIT {limit}"
+    
+    try:
+        result = con.execute(query, params)
+        return result.fetchdf()
+    except Exception as e:
+        st.error(f"Query execution failed: {str(e)}")
+        return pd.DataFrame()
+
+def render_data_grid(df):
+    """
+    Render an interactive data grid (with built‑in filtering) and return the selected row.
+    The grid is configured to show only the desired columns (ID, Date, Source, Tone)
+    and allow filtering/search on each.
+    """
+    st.subheader("Search and Filter Records")
+    
+    # Build grid options with AgGrid
+    gb = GridOptionsBuilder.from_dataframe(df)
+    gb.configure_default_column(filter=True, sortable=True, resizable=True)
+    # Enable single row selection
+    gb.configure_selection('single', use_checkbox=False)
+    grid_options = gb.build()
+
+    # Render AgGrid (the grid will have a filter field for each column)
+    grid_response = AgGrid(
+        df,
+        gridOptions=grid_options,
+        update_mode=GridUpdateMode.SELECTION_CHANGED,
+        height=400,
+        fit_columns_on_grid_load=True
+    )
+    
+    selected = grid_response.get('selected_rows')
+    if selected is not None:
+        # If selected is a DataFrame, use iloc to get the first row.
+        if isinstance(selected, pd.DataFrame):
+            if not selected.empty:
+                return selected.iloc[0].to_dict()
+        # Otherwise, if it's a list, get the first element.
+        elif isinstance(selected, list) and len(selected) > 0:
+            return selected[0]
+    return None
+
+def render_raw_data(record):
+    """Render raw GDELT data in expandable sections."""
+    st.header("Full Record Details")
+    for category, fields in GDELT_CATEGORIES.items():
+        with st.expander(f"{category}"):
+            for field in fields:
+                if field in record:
+                    st.markdown(f"**{field}:**")
+                    st.text(record[field])
+                    st.divider()
+
+def main():
+    st.title("🗺️ COVID Dataset Navigator")
+    st.markdown("""
+    **Explore and Analyze COVID-19 Event Data**
+
+    Use the interactive filters on the sidebar to search, sort, and inspect individual records from the GDELT Global Knowledge Graph. Adjust the parameters below to uncover detailed event insights.
+    """)
+
+    # Initialize database connection using context manager
+    with initialize_db() as con:
+        if con is not None:
+            # Add UI components
+
+            # Sidebar controls
+            with st.sidebar:
+                st.header("Search Filters")
+                source = st.text_input("Filter by source name")
+                start_date = st.text_input("Start date (YYYYMMDD)", "20200314")
+                end_date = st.text_input("End date (YYYYMMDD)", "20200315")
+                limit = st.slider("Number of results to display", 10, 500, 100)
+            
+            # Fetch initial data view
+            df_initial = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=False
+            )
+            
+            # Fetch full records for selection
+            df_full = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=True
+            )
+
+            # Create a DataFrame for the grid with only the key columns
+            grid_df = df_initial[['GKGRECORDID', 'DATE', 'SourceCommonName', 'tone', 'DocumentIdentifier', 'SourceCollectionIdentifier']].copy()
+            grid_df.columns = ['ID', 'Date', 'Source', 'Tone', 'Doc ID', 'Source Collection ID']
+            
+            # Render the interactive data grid at the top
+            selected_row = render_data_grid(grid_df)
+
+            if selected_row:
+                # Find the full record in the original DataFrame using the selected ID
+                selected_id = selected_row['ID']
+                full_record = df_full[df_full['GKGRECORDID'] == selected_id].iloc[0]
+                
+                # Display the raw data below the grid
+                render_raw_data(full_record)
+            else:
+                st.info("Select a record above to view its complete details.")
+        else:
+            st.warning("No matching records found.")
+
+    # Close database connection
+    con.close()
+
+main()

pages/2_🔍_COVID_Event_Graph.py

@@ -0,0 +1,189 @@
+import streamlit as st
+import duckdb
+import pandas as pd
+from st_aggrid import AgGrid, GridOptionsBuilder, GridUpdateMode
+from st_link_analysis import st_link_analysis, NodeStyle, EdgeStyle
+from graph_builder import StLinkBuilder
+
+# Node styles configuration
+NODE_STYLES = [
+    NodeStyle("EVENT", "#FF7F3E", "name", "description"),
+    NodeStyle("PERSON", "#4CAF50", "name", "person"),
+    NodeStyle("NAME", "#2A629A", "created_at", "badge"),
+    NodeStyle("ORGANIZATION", "#9C27B0", "name", "business"),
+    NodeStyle("LOCATION", "#2196F3", "name", "place"),
+    NodeStyle("THEME", "#FFC107", "name", "sell"),
+    NodeStyle("COUNT", "#795548", "name", "inventory"),
+    NodeStyle("AMOUNT", "#607D8B", "name", "wallet"),
+]
+
+# Edge styles configuration
+EDGE_STYLES = [
+    EdgeStyle("MENTIONED_IN", caption="label", directed=True),
+    EdgeStyle("LOCATED_IN", caption="label", directed=True),
+    EdgeStyle("CATEGORIZED_AS", caption="label", directed=True)
+]
+
+def initialize_db():
+    """Initialize database connection and create dataset view"""
+    con = duckdb.connect()
+    con.execute("""
+        CREATE VIEW negative_tone AS (
+            SELECT * 
+            FROM read_parquet('hf://datasets/dwb2023/gdelt-gkg-march2020-v2@~parquet/default/negative_tone/*.parquet')
+        );
+    """)
+    return con
+
+def fetch_data(con, source_filter=None,
+               start_date=None, end_date=None, limit=50, include_all_columns=False):
+    """Fetch filtered data from the database"""
+    if include_all_columns:
+        columns = "*"
+    else:
+        columns = "GKGRECORDID, DATE, SourceCommonName, tone, DocumentIdentifier, 'V2.1Quotations', SourceCollectionIdentifier"
+    
+    query = f"""
+        SELECT {columns}
+        FROM negative_tone 
+        WHERE TRUE
+    """
+    params = []
+    
+    if source_filter:
+        query += " AND SourceCommonName ILIKE ?"
+        params.append(f"%{source_filter}%")
+    if start_date:
+        query += " AND DATE >= ?"
+        params.append(start_date)
+    if end_date:
+        query += " AND DATE <= ?"
+        params.append(end_date)
+    if limit:
+        query += f" LIMIT {limit}"
+    
+    try:
+        result = con.execute(query, params)
+        return result.fetchdf()
+    except Exception as e:
+        st.error(f"Query execution failed: {str(e)}")
+        return pd.DataFrame()
+
+def render_data_grid(df):
+    """
+    Render an interactive data grid (with built‑in filtering) and return the selected row.
+    The grid is configured to show only the desired columns (ID, Date, Source, Tone)
+    and allow filtering/search on each.
+    """
+    st.subheader("Search and Filter Records")
+    
+    # Build grid options with AgGrid
+    gb = GridOptionsBuilder.from_dataframe(df)
+    gb.configure_default_column(filter=True, sortable=True, resizable=True)
+    # Enable single row selection
+    gb.configure_selection('single', use_checkbox=False)
+    grid_options = gb.build()
+
+    # Render AgGrid (the grid will have a filter field for each column)
+    grid_response = AgGrid(
+        df,
+        gridOptions=grid_options,
+        update_mode=GridUpdateMode.SELECTION_CHANGED,
+        height=400,
+        fit_columns_on_grid_load=True
+    )
+    
+    selected = grid_response.get('selected_rows')
+    if selected is not None:
+        # If selected is a DataFrame, use iloc to get the first row.
+        if isinstance(selected, pd.DataFrame):
+            if not selected.empty:
+                return selected.iloc[0].to_dict()
+        # Otherwise, if it's a list, get the first element.
+        elif isinstance(selected, list) and len(selected) > 0:
+            return selected[0]
+    return None
+
+def render_graph(record):
+    """
+    Render a graph visualization for the selected record.
+    Uses StLinkBuilder to convert the record into graph format and then
+    displays the graph using st_link_analysis.
+    """
+    st.subheader(f"Event Graph: {record.get('GKGRECORDID', 'Unknown')}")
+    stlink_builder = StLinkBuilder()
+    # Convert the record (a Series) into a DataFrame with one row
+    record_df = pd.DataFrame([record])
+    graph_data = stlink_builder.build_graph(record_df)
+    return st_link_analysis(
+        elements=graph_data,
+        layout="fcose", # Column configuration for data grid - cose, fcose, breadthfirst, cola
+        node_styles=NODE_STYLES,
+        edge_styles=EDGE_STYLES
+    )
+
+def main():
+    st.title("🔍 COVID Event Graph Explorer")
+    st.markdown("""
+    **Interactive Event Graph Viewer**
+
+    Filter and select individual COVID-19 event records to display their detailed graph representations. Analyze relationships between events and associated entities using the interactive graph below.
+    """)
+
+    # Initialize database connection using context manager
+    with initialize_db() as con:
+        if con is not None:
+            # Add UI components
+
+            # Sidebar controls
+            with st.sidebar:
+                st.header("Search Filters")
+                source = st.text_input("Filter by source name")
+                start_date = st.text_input("Start date (YYYYMMDD)", "20200314")
+                end_date = st.text_input("End date (YYYYMMDD)", "20200315")
+                limit = st.slider("Number of results to display", 10, 500, 100)
+
+            # Fetch initial data view
+            df_initial = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=False
+            )
+            
+            # Fetch full records for selection
+            df_full = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=True
+            )
+            
+            # Create a DataFrame for the grid with only the key columns
+            grid_df = df_initial[['GKGRECORDID', 'DATE', 'SourceCommonName', 'tone', 'DocumentIdentifier', 'SourceCollectionIdentifier']].copy()
+            grid_df.columns = ['ID', 'Date', 'Source', 'Tone', 'Doc ID', 'Source Collection ID']
+            
+            # Render the interactive data grid at the top
+            selected_row = render_data_grid(grid_df)
+            
+            if selected_row:
+                # Find the full record in the original DataFrame using the selected ID
+                selected_id = selected_row['ID']
+                full_record = df_full[df_full['GKGRECORDID'] == selected_id].iloc[0]
+                
+                # Display the graph and raw data below the grid
+                render_graph(full_record)
+            else:
+                st.info("Use the grid filters above to search and select a record.")
+
+        else:
+            st.warning("No matching records found.")
+
+    # Close database connection
+    con.close()
+
+main()

pages/3_🌐_COVID_Network_Analysis.py

@@ -0,0 +1,349 @@
+"""
+Network Analysis Page - GDELT Graph Analysis
+This module provides interactive network analysis of GDELT event data.
+"""
+import streamlit as st
+import networkx as nx
+from pyvis.network import Network
+import pandas as pd
+from datetime import datetime
+import tempfile
+import json
+from typing import Dict, List, Set, Tuple, Optional
+from pathlib import Path
+
+from data_access import get_gdelt_data, filter_dataframe, GDELT_CATEGORIES
+from graph_builder import StreamlitGraphBuilder
+from graph_config import NODE_TYPES
+
+# Type aliases for clarity
+NodeID = str
+CommunityID = int
+Community = Set[NodeID]
+Communities = List[Community]
+
+def create_legend_html() -> str:
+    """Create HTML for the visualization legend."""
+    legend_html = """
+    <div style="
+        position: absolute;
+        top: 10px;
+        right: 10px;
+        background-color: rgba(255, 255, 255, 0.9);
+        padding: 10px;
+        border-radius: 5px;
+        border: 1px solid #ddd;
+        z-index: 1000;
+    ">
+        <h3 style="margin: 0 0 10px 0;">Legend</h3>
+    """
+    
+    for node_type, info in NODE_TYPES.items():
+        legend_html += f"""
+        <div style="margin: 5px 0;">
+            <span style="
+                display: inline-block;
+                width: 12px;
+                height: 12px;
+                background-color: {info['color']};
+                border-radius: 50%;
+                margin-right: 5px;
+            "></span>
+            <span>{info['description']}</span>
+        </div>
+        """
+    
+    legend_html += "</div>"
+    return legend_html
+
+class CommunityAnalyzer:
+    """Handles community detection and analysis for GDELT network graphs."""
+    
+    def __init__(self, G: nx.Graph):
+        self.G = G
+        self._communities: Optional[Communities] = None
+        self._analysis: Optional[List[Dict]] = None
+    
+    @property
+    def communities(self) -> Communities:
+        """Cached access to detected communities."""
+        if self._communities is None:
+            self._communities = nx.community.louvain_communities(self.G)
+        return self._communities
+
+    def analyze_composition(self) -> List[Dict]:
+        """Perform detailed analysis of each community's composition."""
+        if self._analysis is not None:
+            return self._analysis
+            
+        analysis_results = []
+        
+        for idx, community in enumerate(self.communities):
+            try:
+                # Initialize analysis containers
+                node_types = {ntype: 0 for ntype in NODE_TYPES.keys()}
+                themes: Set[str] = set()
+                entities: Dict[str, int] = {}
+                
+                # Analyze community nodes
+                for node in community:
+                    attrs = self.G.nodes[node]
+                    node_type = attrs.get('type', 'unknown')
+                    
+                    # Update type counts
+                    if node_type in node_types:
+                        node_types[node_type] += 1
+                    
+                    # Collect themes
+                    if node_type == 'theme':
+                        theme_name = attrs.get('name', '')
+                        if theme_name:
+                            themes.add(theme_name)
+                    
+                    # Track entity connections
+                    if node_type in {'person', 'organization', 'location'}:
+                        name = attrs.get('name', node)
+                        entities[name] = self.G.degree(node)
+                
+                # Calculate community metrics
+                subgraph = self.G.subgraph(community)
+                n = len(community)
+                possible_edges = (n * (n - 1)) / 2 if n > 1 else 0
+                density = (subgraph.number_of_edges() / possible_edges) if possible_edges > 0 else 0
+                
+                # Get top entities by degree
+                top_entities = dict(sorted(entities.items(), key=lambda x: x[1], reverse=True)[:5])
+                
+                analysis_results.append({
+                    'id': idx,
+                    'size': len(community),
+                    'node_types': node_types,
+                    'themes': sorted(themes),
+                    'top_entities': top_entities,
+                    'density': density,
+                    'internal_edges': subgraph.number_of_edges(),
+                    'external_edges': sum(1 for u in community 
+                                        for v in self.G[u] 
+                                        if v not in community)
+                })
+                
+            except Exception as e:
+                st.error(f"Error analyzing community {idx}: {str(e)}")
+                continue
+        
+        self._analysis = analysis_results
+        return analysis_results
+
+def display_community_analysis(analysis: List[Dict]) -> None:
+    """Display detailed community analysis in Streamlit."""
+    # Display summary metrics
+    total_nodes = sum(comm['size'] for comm in analysis)
+    col1, col2, col3 = st.columns(3)
+    with col1:
+        st.metric("Total Communities", len(analysis))
+    with col2:
+        st.metric("Total Nodes", total_nodes)
+    with col3:
+        largest_comm = max(comm['size'] for comm in analysis)
+        st.metric("Largest Community", largest_comm)
+    
+    # Display each community in tabs
+    st.subheader("Community Details")
+    tabs = st.tabs([f"Community {comm['id']}" for comm in analysis])
+    for tab, comm in zip(tabs, analysis):
+        with tab:
+            cols = st.columns(2)
+            
+            # Left column: Composition
+            with cols[0]:
+                st.subheader("Composition")
+                node_types_df = pd.DataFrame([comm['node_types']]).T
+                node_types_df.columns = ['Count']
+                st.bar_chart(node_types_df)
+                
+                st.markdown("**Metrics:**")
+                st.write(f"- Size: {comm['size']} nodes")
+                st.write(f"- Density: {comm['density']:.3f}")
+                st.write(f"- Internal edges: {comm['internal_edges']}")
+                st.write(f"- External edges: {comm['external_edges']}")
+                st.write(f"- % of network: {(comm['size']/total_nodes)*100:.1f}%")
+            
+            # Right column: Entities and Themes
+            with cols[1]:
+                if comm['top_entities']:
+                    st.subheader("Key Entities")
+                    for entity, degree in comm['top_entities'].items():
+                        st.write(f"- {entity} ({degree} connections)")
+                
+                if comm['themes']:
+                    st.subheader("Themes")
+                    for theme in sorted(comm['themes']):
+                        st.write(f"- {theme}")
+
+def visualize_with_pyvis(G: nx.Graph, physics: bool = True) -> str:
+    """Create interactive PyVis visualization with legend."""
+    net = Network(height="600px", width="100%", notebook=False, directed=False)
+    net.from_nx(G)
+    
+    # Configure nodes
+    for node in net.nodes:
+        node_type = node.get("type", "unknown")
+        node["color"] = NODE_TYPES.get(node_type, {}).get('color', "#cccccc")
+        node["size"] = 20 if node_type == "event" else 15
+        title_attrs = {k: v for k, v in node.items() if k != "id"}
+        node["title"] = "<br>".join(f"{k}: {v}" for k, v in title_attrs.items())
+
+    # Configure edges
+    for edge in net.edges:
+        edge["title"] = edge.get("relationship", "")
+        edge["color"] = {"color": "#666666", "opacity": 0.5}
+    
+    # Physics settings
+    if physics:
+        net.show_buttons(filter_=['physics'])
+    else:
+        net.toggle_physics(False)
+    
+    # Generate HTML
+    with tempfile.NamedTemporaryFile(delete=False, suffix=".html") as f:
+        net.save_graph(f.name)
+        html_content = Path(f.name).read_text(encoding='utf-8')
+    
+    # Add legend
+    legend = create_legend_html()
+    html_content = html_content.replace('</body>', f'{legend}</body>')
+    
+    return html_content
+
+def main():
+    st.title("🌐 Global Network Analysis")
+    st.markdown("""
+    **Explore Global Event Networks**
+
+    Dive deep into the interconnected world of negative sentiment events as captured by GDELT. Utilize interactive visualizations and community analysis tools to understand key metrics, structures, and interrelationships.
+    """)
+
+
+    # Initialize session state
+    if 'vis_html' not in st.session_state:
+        st.session_state.vis_html = None
+   
+    # Sidebar controls
+    with st.sidebar:
+        st.header("Graph Controls")
+        limit = st.slider("Max records to load", 1, 25, 5)
+        tone_threshold = st.slider("Max tone score", -10.0, -5.0, -7.0)
+        show_physics = st.checkbox("Enable physics", value=True)
+    
+        st.header("Advanced Filters")
+        source_filter = st.text_input("Filter by source name")
+        themes_filter = st.text_input("Filter by theme/keyword")
+        start_date = st.text_input("Start date (YYYYMMDD)")
+        end_date = st.text_input("End date (YYYYMMDD)")
+
+    try:
+        # Load and process data
+        df = get_gdelt_data(
+            limit=limit,
+            tone_threshold=tone_threshold,
+            start_date=start_date if start_date else None,
+            end_date=end_date if end_date else None,
+            source_filter=source_filter,
+            themes_filter=themes_filter
+        )
+        
+        # Build graph
+        with st.spinner("Building knowledge graph..."):
+            builder = StreamlitGraphBuilder()
+            for _, row in df.iterrows():
+                builder.process_row(row)
+            G = builder.G
+        
+        if G.number_of_nodes() == 0:
+            st.warning("No data found matching the specified criteria.")
+            return
+        
+        # Display basic metrics
+        col1, col2, col3 = st.columns(3)
+        with col1:
+            st.metric("Total Nodes", G.number_of_nodes())
+        with col2:
+            st.metric("Total Edges", G.number_of_edges())
+        with col3:
+            event_count = sum(1 for _, attr in G.nodes(data=True) 
+                            if attr.get("type") == "event")
+            st.metric("Negative Events", event_count)
+        
+        # Analysis section
+        st.header("NetworkX Graph Analysis")
+        
+        # Centrality analysis
+        with st.expander("Centrality Analysis"):
+            degree_centrality = nx.degree_centrality(G)
+            top_nodes = sorted(degree_centrality.items(), 
+                             key=lambda x: x[1], reverse=True)[:5]
+            
+            st.write("Most Connected Nodes:")
+            for node, centrality in top_nodes:
+                node_type = G.nodes[node].get("type", "unknown")
+                st.write(f"- `{node[:30]}` ({node_type}): {centrality:.3f}")
+        
+        # Community analysis
+        with st.expander("Community Analysis"):
+            try:
+                analyzer = CommunityAnalyzer(G)
+                analysis = analyzer.analyze_composition()
+                display_community_analysis(analysis)
+            except Exception as e:
+                st.error(f"Community analysis failed: {str(e)}")
+                st.error("Please check the graph structure and try again.")
+            
+        # Export options
+        st.header("Export Options")
+        with st.expander("Export Data"):
+            col1, col2, col3 = st.columns(3)
+            
+            with col1:
+                # GraphML export
+                graphml_string = "".join(nx.generate_graphml(G))
+                st.download_button(
+                    label="Download GraphML",
+                    data=graphml_string.encode('utf-8'),
+                    file_name=f"gdelt_graph_{datetime.now().isoformat()}.graphml",
+                    mime="application/xml"
+                )
+            
+            with col2:
+                # JSON network export
+                json_string = json.dumps(nx.node_link_data(G, edges="edges"))
+                st.download_button(
+                    label="Download JSON",
+                    data=json_string.encode('utf-8'),
+                    file_name=f"gdelt_graph_{datetime.now().isoformat()}.json",
+                    mime="application/json"
+                )
+            
+            with col3:
+                # Community analysis export
+                if 'analysis' in locals():
+                    analysis_json = json.dumps(analysis, indent=2)
+                    st.download_button(
+                        label="Download Analysis",
+                        data=analysis_json.encode('utf-8'),
+                        file_name=f"community_analysis_{datetime.now().isoformat()}.json",
+                        mime="application/json"
+                    )
+        
+        # Interactive visualization
+        st.header("Network Visualization")
+        with st.expander("Interactive Network", expanded=False):
+            if st.session_state.vis_html is None:
+                with st.spinner("Generating visualization..."):
+                    st.session_state.vis_html = visualize_with_pyvis(G, physics=show_physics)
+            st.components.v1.html(st.session_state.vis_html, height=600, scrolling=True)
+    
+    except Exception as e:
+        st.error(f"An error occurred: {str(e)}")
+        st.error("Please adjust your filters and try again.")
+
+main()

pages/4_🗺️_Feb_2025_Navigator.py

@@ -0,0 +1,185 @@
+import streamlit as st
+import duckdb
+import pandas as pd
+from st_aggrid import AgGrid, GridOptionsBuilder, GridUpdateMode
+
+# Constants for raw data categories
+GDELT_CATEGORIES = {
+    "Metadata": ["GKGRECORDID", "DATE", "SourceCommonName", "DocumentIdentifier", "V2.1Quotations", "tone"],
+    "Persons": ["V2EnhancedPersons", "V1Persons"],
+    "Organizations": ["V2EnhancedOrganizations", "V1Organizations"],
+    "Locations": ["V2EnhancedLocations", "V1Locations"],
+    "Themes": ["V2EnhancedThemes", "V1Themes"],
+    "Names": ["V2.1AllNames"],
+    "Counts": ["V2.1Counts", "V1Counts"],
+    "Amounts": ["V2.1Amounts"],
+    "V2GCAM": ["V2GCAM"],
+    "V2.1EnhancedDates": ["V2.1EnhancedDates"],
+}
+
+def initialize_db():
+    """Initialize database connection and create dataset view with optimized tone extraction"""
+    con = duckdb.connect()
+    con.execute("""
+        CREATE VIEW tone_vw AS (
+            SELECT 
+                * EXCLUDE ("V1.5Tone"),
+                TRY_CAST(
+                    CASE 
+                        WHEN POSITION(',' IN "V1.5Tone") > 0 
+                        THEN SUBSTRING("V1.5Tone", 1, POSITION(',' IN "V1.5Tone") - 1)
+                        ELSE "V1.5Tone"
+                    END
+                    AS FLOAT
+                ) AS tone 
+            FROM read_parquet('hf://datasets/dwb2023/gdelt-gkg-2025-v2/**/*.parquet')
+        );
+    """)
+    return con
+
+def fetch_data(con, source_filter=None,
+               start_date=None, end_date=None, limit=10, include_all_columns=False):
+    """Fetch filtered data from the database"""
+    if include_all_columns:
+        columns = "*"
+    else:
+        # Changed column specification: use double quotes for column names with periods.
+        columns = 'GKGRECORDID, DATE, SourceCommonName, tone, DocumentIdentifier, "V2.1SharingImage", "V2.1Quotations", SourceCollectionIdentifier'
+    
+    query = f"""
+        SELECT {columns}
+        FROM tone_vw 
+        WHERE TRUE
+    """
+    params = []
+    
+    if source_filter:
+        query += " AND SourceCommonName ILIKE ?"
+        params.append(f"%{source_filter}%")
+    if start_date:
+        query += " AND DATE >= ?"
+        params.append(start_date)
+    if end_date:
+        query += " AND DATE <= ?"
+        params.append(end_date)
+    if limit:
+        query += f" LIMIT {limit}"
+    
+    try:
+        result = con.execute(query, params)
+        return result.fetchdf()
+    except Exception as e:
+        st.error(f"Query execution failed: {str(e)}")
+        return pd.DataFrame()
+
+def render_data_grid(df):
+    """
+    Render an interactive data grid (with built‑in filtering) and return the selected row.
+    The grid is configured to show only the desired columns (ID, Date, Source, Tone)
+    and allow filtering/search on each.
+    """
+    st.subheader("Search and Filter Records")
+    
+    # Build grid options with AgGrid
+    gb = GridOptionsBuilder.from_dataframe(df)
+    gb.configure_default_column(filter=True, sortable=True, resizable=True)
+    # Enable single row selection
+    gb.configure_selection('single', use_checkbox=False)
+    grid_options = gb.build()
+
+    # Render AgGrid (the grid will have a filter field for each column)
+    grid_response = AgGrid(
+        df,
+        gridOptions=grid_options,
+        update_mode=GridUpdateMode.SELECTION_CHANGED,
+        height=400,
+        fit_columns_on_grid_load=True
+    )
+    
+    selected = grid_response.get('selected_rows')
+    if selected is not None:
+        # If selected is a DataFrame, use iloc to get the first row.
+        if isinstance(selected, pd.DataFrame):
+            if not selected.empty:
+                return selected.iloc[0].to_dict()
+        # Otherwise, if it's a list, get the first element.
+        elif isinstance(selected, list) and len(selected) > 0:
+            return selected[0]
+    return None
+
+def render_raw_data(record):
+    """Render raw GDELT data in expandable sections."""
+    st.header("Full Record Details")
+    for category, fields in GDELT_CATEGORIES.items():
+        with st.expander(f"{category}"):
+            for field in fields:
+                if field in record:
+                    st.markdown(f"**{field}:**")
+                    st.text(record[field])
+                    st.divider()
+
+def main():
+    st.title("🗺️ GDELT Feb 2025 Navigator")
+    st.markdown("""
+    **Investigate Recent Global Events (Feb 2025)**
+
+    Leverage advanced filters and interactive grids to explore the latest data from the GDELT Global Knowledge Graph. This navigator is optimized for recent events, offering insights into evolving global narratives.
+    """)
+
+
+    # Initialize database connection using context manager
+    with initialize_db() as con:
+        if con is not None:
+            # Add UI components
+
+            # Sidebar controls
+            with st.sidebar:
+                st.header("Search Filters")
+                source = st.text_input("Filter by source name")
+                start_date = st.text_input("Start date (YYYYMMDD)", "20250210")
+                end_date = st.text_input("End date (YYYYMMDD)", "20250211")
+                limit = st.slider("Number of results to display", 10, 500, 10)
+            
+            # Fetch initial data view
+            df_initial = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=False
+            )
+            
+            # Fetch full records for selection
+            df_full = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=True
+            )
+
+            # Create a DataFrame for the grid with only the key columns
+            grid_df = df_initial[['GKGRECORDID', 'DATE', 'SourceCommonName', 'tone', 'DocumentIdentifier',  "V2.1SharingImage", 'SourceCollectionIdentifier']].copy()
+            grid_df.columns = ['ID', 'Date', 'Source', 'Tone', 'Doc ID', 'Image', 'Source Collection ID']
+            
+            # Render the interactive data grid at the top
+            selected_row = render_data_grid(grid_df)
+
+            if selected_row:
+                # Find the full record in the original DataFrame using the selected ID
+                selected_id = selected_row['ID']
+                full_record = df_full[df_full['GKGRECORDID'] == selected_id].iloc[0]
+                
+                # Display the raw data below the grid
+                render_raw_data(full_record)
+            else:
+                st.info("Select a record above to view its complete details.")
+        else:
+            st.warning("No matching records found.")
+
+    # Close database connection
+    con.close()
+
+main()

pages/5_🔍_Feb_2025_Event_Graph.py

@@ -0,0 +1,198 @@
+import streamlit as st
+import duckdb
+import pandas as pd
+from st_aggrid import AgGrid, GridOptionsBuilder, GridUpdateMode
+from st_link_analysis import st_link_analysis, NodeStyle, EdgeStyle
+from graph_builder import StLinkBuilder
+
+# Node styles configuration
+NODE_STYLES = [
+    NodeStyle("EVENT", "#FF7F3E", "name", "description"),
+    NodeStyle("PERSON", "#4CAF50", "name", "person"),
+    NodeStyle("NAME", "#2A629A", "created_at", "badge"),
+    NodeStyle("ORGANIZATION", "#9C27B0", "name", "business"),
+    NodeStyle("LOCATION", "#2196F3", "name", "place"),
+    NodeStyle("THEME", "#FFC107", "name", "sell"),
+    NodeStyle("COUNT", "#795548", "name", "inventory"),
+    NodeStyle("AMOUNT", "#607D8B", "name", "wallet"),
+]
+
+# Edge styles configuration
+EDGE_STYLES = [
+    EdgeStyle("MENTIONED_IN", caption="label", directed=True),
+    EdgeStyle("LOCATED_IN", caption="label", directed=True),
+    EdgeStyle("CATEGORIZED_AS", caption="label", directed=True)
+]
+
+def initialize_db():
+    """Initialize database connection and create dataset view with optimized tone extraction"""
+    con = duckdb.connect()
+    con.execute("""
+        CREATE VIEW tone_vw AS (
+            SELECT 
+                * EXCLUDE ("V1.5Tone"),
+                TRY_CAST(
+                    CASE 
+                        WHEN POSITION(',' IN "V1.5Tone") > 0 
+                        THEN SUBSTRING("V1.5Tone", 1, POSITION(',' IN "V1.5Tone") - 1)
+                        ELSE "V1.5Tone"
+                    END
+                    AS FLOAT
+                ) AS tone 
+            FROM read_parquet('hf://datasets/dwb2023/gdelt-gkg-2025-v2/**/*.parquet')
+        );
+    """)
+    return con
+
+def fetch_data(con, source_filter=None,
+               start_date=None, end_date=None, limit=50, include_all_columns=False):
+    """Fetch filtered data from the database"""
+    if include_all_columns:
+        columns = "*"
+    else:
+        columns = "GKGRECORDID, DATE, SourceCommonName, tone, DocumentIdentifier, 'V2.1Quotations', SourceCollectionIdentifier"
+    
+    query = f"""
+        SELECT {columns}
+        FROM tone_vw 
+        WHERE TRUE
+    """
+    params = []
+    
+    if source_filter:
+        query += " AND SourceCommonName ILIKE ?"
+        params.append(f"%{source_filter}%")
+    if start_date:
+        query += " AND DATE >= ?"
+        params.append(start_date)
+    if end_date:
+        query += " AND DATE <= ?"
+        params.append(end_date)
+    if limit:
+        query += f" LIMIT {limit}"
+    
+    try:
+        result = con.execute(query, params)
+        return result.fetchdf()
+    except Exception as e:
+        st.error(f"Query execution failed: {str(e)}")
+        return pd.DataFrame()
+
+def render_data_grid(df):
+    """
+    Render an interactive data grid (with built‑in filtering) and return the selected row.
+    The grid is configured to show only the desired columns (ID, Date, Source, Tone)
+    and allow filtering/search on each.
+    """
+    st.subheader("Search and Filter Records")
+    
+    # Build grid options with AgGrid
+    gb = GridOptionsBuilder.from_dataframe(df)
+    gb.configure_default_column(filter=True, sortable=True, resizable=True)
+    # Enable single row selection
+    gb.configure_selection('single', use_checkbox=False)
+    grid_options = gb.build()
+
+    # Render AgGrid (the grid will have a filter field for each column)
+    grid_response = AgGrid(
+        df,
+        gridOptions=grid_options,
+        update_mode=GridUpdateMode.SELECTION_CHANGED,
+        height=400,
+        fit_columns_on_grid_load=True
+    )
+    
+    selected = grid_response.get('selected_rows')
+    if selected is not None:
+        # If selected is a DataFrame, use iloc to get the first row.
+        if isinstance(selected, pd.DataFrame):
+            if not selected.empty:
+                return selected.iloc[0].to_dict()
+        # Otherwise, if it's a list, get the first element.
+        elif isinstance(selected, list) and len(selected) > 0:
+            return selected[0]
+    return None
+
+def render_graph(record):
+    """
+    Render a graph visualization for the selected record.
+    Uses StLinkBuilder to convert the record into graph format and then
+    displays the graph using st_link_analysis.
+    """
+    st.subheader(f"Event Graph: {record.get('GKGRECORDID', 'Unknown')}")
+    stlink_builder = StLinkBuilder()
+    # Convert the record (a Series) into a DataFrame with one row
+    record_df = pd.DataFrame([record])
+    graph_data = stlink_builder.build_graph(record_df)
+    return st_link_analysis(
+        elements=graph_data,
+        layout="fcose", # Column configuration for data grid - cose, fcose, breadthfirst, cola
+        node_styles=NODE_STYLES,
+        edge_styles=EDGE_STYLES
+    )
+
+def main():
+    st.title("🔍 GDELT Feb 2025 Event Graph Explorer")
+    st.markdown("""
+    **Investigate Recent Global Events (Feb 2025) in an Interactive Event Graph Viewer**
+
+    Filter and select individual event records to display their detailed graph representations. Analyze relationships between events and associated entities using the interactive graph below.
+    """)
+
+    # Initialize database connection using context manager
+    with initialize_db() as con:
+        if con is not None:
+            # Add UI components
+
+            # Sidebar controls
+            with st.sidebar:
+                st.header("Search Filters")
+                source = st.text_input("Filter by source name")
+                start_date = st.text_input("Start date (YYYYMMDD)", "20250210")
+                end_date = st.text_input("End date (YYYYMMDD)", "20250211")
+                limit = st.slider("Number of results to display", 10, 500, 100)
+
+            # Fetch initial data view
+            df_initial = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=False
+            )
+            
+            # Fetch full records for selection
+            df_full = fetch_data(
+                con=con,
+                source_filter=source,
+                start_date=start_date,
+                end_date=end_date,
+                limit=limit,
+                include_all_columns=True
+            )
+            
+            # Create a DataFrame for the grid with only the key columns
+            grid_df = df_initial[['GKGRECORDID', 'DATE', 'SourceCommonName', 'tone', 'DocumentIdentifier', 'SourceCollectionIdentifier']].copy()
+            grid_df.columns = ['ID', 'Date', 'Source', 'Tone', 'Doc ID', 'Source Collection ID']
+            
+            # Render the interactive data grid at the top
+            selected_row = render_data_grid(grid_df)
+            
+            if selected_row:
+                # Find the full record in the original DataFrame using the selected ID
+                selected_id = selected_row['ID']
+                full_record = df_full[df_full['GKGRECORDID'] == selected_id].iloc[0]
+                
+                # Display the graph and raw data below the grid
+                render_graph(full_record)
+            else:
+                st.info("Use the grid filters above to search and select a record.")
+
+        else:
+            st.warning("No matching records found.")
+
+    # Close database connection
+    con.close()
+
+main()

pages/6_🧪_Feb_2025_Dataset_Explorer.py

@@ -0,0 +1,250 @@
+import streamlit as st
+import pandas as pd
+from datasets import load_dataset
+import re
+from datetime import datetime, date
+from io import StringIO
+from typing import Optional, Tuple, List, Dict, Any
+
+# Constants
+DEFAULT_SAMPLE_SIZE = 1000
+DATE_FORMAT = "%Y%m%d"
+FULL_DATE_FORMAT = f"{DATE_FORMAT}%H%M%S"
+
+# Load dataset with enhanced caching and validation
+@st.cache_data(ttl=3600, show_spinner="Loading dataset...")
+def load_data(sample_size: int = DEFAULT_SAMPLE_SIZE) -> pd.DataFrame:
+    """
+    Load and validate dataset with error handling.
+    
+    Args:
+        sample_size (int): Number of records to load
+        
+    Returns:
+        pd.DataFrame: Loaded and validated dataframe
+    """
+    try:
+        dataset = load_dataset(
+            "dwb2023/gdelt-gkg-2025-v2",
+            data_files={
+                "train": [
+                    "gdelt_gkg_20250210.parquet",
+                    "gdelt_gkg_20250211.parquet",
+                ]
+            },
+            split="train"
+        )
+        df = pd.DataFrame(dataset)
+
+        # Basic data validation
+        if df.empty:
+            st.error("Loaded dataset is empty")
+            return pd.DataFrame()
+            
+        if "DATE" not in df.columns:
+            st.error("Dataset missing required DATE column")
+            return pd.DataFrame()
+            
+        return df
+        
+    except Exception as e:
+        st.error(f"Error loading dataset: {str(e)}")
+        st.stop()
+        return pd.DataFrame()
+
+def initialize_app(df: pd.DataFrame) -> None:
+    """Initialize the Streamlit app interface."""
+    st.title("GDELT GKG 2025 Dataset Explorer")
+    
+    with st.sidebar:
+        st.header("Search Criteria")
+        st.markdown("🔍 Filter dataset using the controls below")
+
+def extract_unique_themes(df: pd.DataFrame, column: str) -> List[str]:
+    """
+    Extract and clean unique themes from semicolon-separated column.
+    
+    Args:
+        df (pd.DataFrame): Input dataframe
+        column (str): Column name containing themes
+        
+    Returns:
+        List[str]: Sorted list of unique themes
+    """
+    if df.empty:
+        return []
+        
+    return sorted({
+        theme.split(",")[0].strip()
+        for themes in df[column].dropna().str.split(";")
+        for theme in themes if theme.strip()
+    })
+
+def get_date_range(df: pd.DataFrame, date_col: str) -> Tuple[date, date]:
+    """
+    Get min/max dates from dataset with fallback defaults.
+    
+    Args:
+        df (pd.DataFrame): Input dataframe
+        date_col (str): Column name containing dates
+        
+    Returns:
+        Tuple[date, date]: (min_date, max_date) as date objects
+    """
+    try:
+        # Convert YYYYMMDDHHMMSS string format to datetime using constant
+        dates = pd.to_datetime(df[date_col], format=FULL_DATE_FORMAT)
+        return dates.min().date(), dates.max().date()
+    except Exception as e:
+        st.warning(f"Date range detection failed: {str(e)}")
+        return datetime(2025, 2, 10).date(), datetime(2025, 2, 11).date()
+
+def create_filters(df: pd.DataFrame) -> Dict[str, Any]:
+    """
+    Generate sidebar filters and return filter state.
+    
+    Args:
+        df (pd.DataFrame): Input dataframe
+        
+    Returns:
+        Dict[str, Any]: Dictionary of filter settings
+    """
+    filters = {}
+    
+    with st.sidebar:
+        # Theme multi-select
+        filters["themes"] = st.multiselect(
+            "V2EnhancedThemes (exact match)",
+            options=extract_unique_themes(df, "V2EnhancedThemes"),
+            help="Select exact themes to include (supports multiple selection)"
+        )
+
+        # Text-based filters
+        text_filters = {
+            "source_common_name": ("SourceCommonName", "partial name match"),
+            "document_identifier": ("DocumentIdentifier", "partial identifier match"),
+            "sharing_image": ("V2.1SharingImage", "partial image URL match")
+        }
+        
+        for key, (label, help_text) in text_filters.items():
+            filters[key] = st.text_input(
+                f"{label} ({help_text})",
+                placeholder=f"Enter {help_text}...",
+                help=f"Case-insensitive {help_text}"
+            )
+
+        # Date range with dataset-based defaults
+        date_col = "DATE"
+        min_date, max_date = get_date_range(df, date_col)
+        
+        filters["date_range"] = st.date_input(
+            "Date range",
+            value=(min_date, max_date),
+            min_value=min_date,
+            max_value=max_date,
+        )
+
+        # Record limit
+        filters["record_limit"] = st.number_input(
+            "Max records to display",
+            min_value=100,
+            max_value=5000,
+            value=1000,
+            step=100,
+            help="Limit results for better performance"
+        )
+    
+    return filters
+
+def apply_filters(df: pd.DataFrame, filters: Dict[str, Any]) -> pd.DataFrame:
+    """
+    Apply all filters to dataframe using vectorized operations.
+    
+    Args:
+        df (pd.DataFrame): Input dataframe to filter
+        filters (Dict[str, Any]): Dictionary containing filter parameters:
+            - themes (list): List of themes to match exactly
+            - source_common_name (str): Partial match for source name
+            - document_identifier (str): Partial match for document ID
+            - sharing_image (str): Partial match for image URL
+            - date_range (tuple): (start_date, end_date) tuple
+            - record_limit (int): Maximum number of records to return
+            
+    Returns:
+        pd.DataFrame: Filtered dataframe
+    """
+    filtered_df = df.copy()
+
+    # Theme exact match filter - set regex groups to be non-capturing using (?:) syntax
+    if filters["themes"]:
+        pattern = r'(?:^|;)(?:{})(?:$|,|;)'.format('|'.join(map(re.escape, filters["themes"])))
+        filtered_df = filtered_df[filtered_df["V2EnhancedThemes"].str.contains(pattern, na=False)]
+
+    # Text partial match filters
+    text_columns = {
+        "source_common_name": "SourceCommonName",
+        "document_identifier": "DocumentIdentifier",
+        "sharing_image": "V2.1SharingImage"
+    }
+    
+    for filter_key, col_name in text_columns.items():
+        if value := filters.get(filter_key):
+            filtered_df = filtered_df[
+                filtered_df[col_name]
+                .str.contains(re.escape(value), case=False, na=False)
+            ]
+
+    # Date range filter with validation
+    if len(filters["date_range"]) == 2:
+        start_date, end_date = filters["date_range"]
+        
+        # Validate date range
+        if start_date > end_date:
+            st.error("Start date must be before end date")
+            return filtered_df
+            
+        date_col = "DATE"
+        try:
+            # Convert full datetime strings to datetime objects using constant
+            date_series = pd.to_datetime(filtered_df[date_col], format=FULL_DATE_FORMAT)
+            
+            # Create timestamps for start/end of day
+            start_timestamp = pd.Timestamp(start_date).normalize()  # Start of day
+            end_timestamp = pd.Timestamp(end_date) + pd.Timedelta(days=1) - pd.Timedelta(seconds=1)  # End of day
+            
+            filtered_df = filtered_df[
+                (date_series >= start_timestamp) &
+                (date_series <= end_timestamp)
+            ]
+        except Exception as e:
+            st.error(f"Error applying date filter: {str(e)}")
+            return filtered_df
+
+    # Apply record limit
+    return filtered_df.head(filters["record_limit"])
+
+def main():
+    """Main application entry point."""
+    df = load_data()
+    if df.empty:
+        st.warning("No data available - check data source")
+        return
+        
+    initialize_app(df)
+    filters = create_filters(df)
+    filtered_df = apply_filters(df, filters)
+    
+    # Display results
+    st.subheader(f"Results: {len(filtered_df)} records")
+    
+    st.dataframe(filtered_df, use_container_width=True)
+    
+    st.download_button(
+        label="Download CSV",
+        data=filtered_df.to_csv(index=False).encode(),
+        file_name="filtered_results.csv",
+        mime="text/csv",
+        help="Download filtered results as CSV"
+    )
+
+main()

requirements.txt

@@ -0,0 +1,10 @@
+streamlit
+duckdb
+networkx
+pandas
+pyvis
+datasets
+huggingface_hub
+python-dateutil
+st-link-analysis
+streamlit-aggrid

solution_component_notes/gdelt_gkg_duckdb_networkx_v5.ipynb

@@ -0,0 +1,388 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "N20l3SsqSVUM"
+      },
+      "source": [
+        "## Leveraging DuckDB with HF Datasets - GDELT Global KG\n",
+        "\n",
+        "This notebook demonstrates how to seamlessly transform **GDELT** knowledge graph data into a coherent format that can be pushed to both **NetworkX** and **Neo4j**. It provides a **referenceable pipeline** for data professionals, researchers, and solution architects who need to:\n",
+        "\n",
+        "1. **Ingest and Query Data Efficiently**  \n",
+        "   - Utilize **DuckDB** to load just the required portions of large Parquet datasets, enabling targeted data exploration and analysis.\n",
+        "   - It also allows for iteratively honing in on a specific segment of data using splits - helping to maximize performance / cost / efficiency.\n",
+        "\n",
+        "2. **Maintain Consistent Graph Modeling**  \n",
+        "   - Leverage a shared parsing and entity extraction layer to build consistent node and relationship structures in both an **in-memory** graph (NetworkX) and a **Neo4j** database. (not a requirement per se - but an approach I wanted to start with)\n",
+        "\n",
+        "3. **Run Advanced Queries and Analytics**  \n",
+        "   - Illustrate critical tasks like **centrality** and **community detection** to pinpoint influential nodes and groupings, and execute **Cypher** queries for real-time insights.\n",
+        "\n",
+        "4. **Visualize and Export**  \n",
+        "   - Produce simple web-based **PyVis** visualizations or **matplotlib** plots.\n",
+        "   - more importantly the data can also be exported in **JSON** and GraphML for integration with other graph tooling. (D3.js, Cytoscape, etc.)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "DCPEB5tpfW44"
+      },
+      "outputs": [],
+      "source": [
+        "%pip install -q duckdb networkx pandas neo4j pyvis"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "A1vEyOkm7LPV"
+      },
+      "outputs": [],
+      "source": [
+        "from google.colab import userdata\n",
+        "\n",
+        "URI = userdata.get('NEO4J_URI')\n",
+        "USER = 'neo4j'\n",
+        "PASSWORD = userdata.get('NEO4J_PASSWORD')"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "cm8t66uPy_C7"
+      },
+      "outputs": [],
+      "source": [
+        "import duckdb\n",
+        "import networkx as nx\n",
+        "from neo4j import GraphDatabase\n",
+        "import logging\n",
+        "from datetime import datetime\n",
+        "import pandas as pd\n",
+        "from pyvis.network import Network\n",
+        "\n",
+        "def get_gdelt_data(limit=100):\n",
+        "    \"\"\"Get data from DuckDB with specified limit\"\"\"\n",
+        "    con = duckdb.connect(database=':memory:')\n",
+        "\n",
+        "    # Create view of the dataset\n",
+        "    con.execute(\"\"\"\n",
+        "        CREATE VIEW train AS (\n",
+        "            SELECT *\n",
+        "            FROM read_parquet('hf://datasets/dwb2023/gdelt-gkg-march2020-v2/*.parquet')\n",
+        "        );\n",
+        "    \"\"\")\n",
+        "\n",
+        "    # Single query with limit\n",
+        "    query = f\"\"\"\n",
+        "    SELECT\n",
+        "        GKGRECORDID,\n",
+        "        DATE,\n",
+        "        SourceCommonName,\n",
+        "        DocumentIdentifier,\n",
+        "        V2EnhancedPersons,\n",
+        "        V2EnhancedOrganizations,\n",
+        "        V2EnhancedLocations,\n",
+        "        V2EnhancedThemes,\n",
+        "        CAST(SPLIT_PART(\"V1.5Tone\", ',', 1) AS FLOAT) as tone\n",
+        "    FROM train\n",
+        "    LIMIT {limit}\n",
+        "    \"\"\"\n",
+        "\n",
+        "    results_df = con.execute(query).fetchdf()\n",
+        "    con.close()\n",
+        "    return results_df\n",
+        "\n",
+        "class GraphBuilder:\n",
+        "    \"\"\"Base class for building graph from GDELT data\"\"\"\n",
+        "    def process_entities(self, row):\n",
+        "        \"\"\"Process entities from a row and return nodes and relationships\"\"\"\n",
+        "        nodes = []\n",
+        "        relationships = []\n",
+        "        event_id = row[\"GKGRECORDID\"]\n",
+        "        event_date = row[\"DATE\"]\n",
+        "        event_source = row[\"SourceCommonName\"]\n",
+        "        event_document_id = row[\"DocumentIdentifier\"]\n",
+        "        event_tone = float(row[\"tone\"]) if pd.notna(row[\"tone\"]) else 0.0\n",
+        "\n",
+        "        # Add event node\n",
+        "        nodes.append({\n",
+        "            \"id\": event_id,\n",
+        "            \"type\": \"event\",\n",
+        "            \"properties\": {\n",
+        "                \"date\": event_date,\n",
+        "                \"source\": event_source,\n",
+        "                \"document\": event_document_id,\n",
+        "                \"tone\": event_tone\n",
+        "            }\n",
+        "        })\n",
+        "\n",
+        "        # Process each entity type\n",
+        "        entity_mappings = {\n",
+        "            \"V2EnhancedPersons\": (\"Person\", \"MENTIONED_IN\"),\n",
+        "            \"V2EnhancedOrganizations\": (\"Organization\", \"MENTIONED_IN\"),\n",
+        "            \"V2EnhancedLocations\": (\"Location\", \"LOCATED_IN\"),\n",
+        "            \"V2EnhancedThemes\": (\"Theme\", \"CATEGORIZED_AS\")\n",
+        "        }\n",
+        "\n",
+        "        for field, (label, relationship) in entity_mappings.items():\n",
+        "            if pd.notna(row[field]):\n",
+        "                entities = [e.strip() for e in row[field].split(';') if e.strip()]\n",
+        "                for entity in entities:\n",
+        "                    nodes.append({\n",
+        "                        \"id\": entity,\n",
+        "                        \"type\": label.lower(),\n",
+        "                        \"properties\": {\"name\": entity}\n",
+        "                    })\n",
+        "                    relationships.append({\n",
+        "                        \"from\": entity,\n",
+        "                        \"to\": event_id,\n",
+        "                        \"type\": relationship,\n",
+        "                        \"properties\": {\"created_at\": event_date}\n",
+        "                    })\n",
+        "\n",
+        "        return nodes, relationships\n",
+        "\n",
+        "class NetworkXBuilder(GraphBuilder):\n",
+        "    def build_graph(self, df):\n",
+        "        G = nx.Graph()\n",
+        "\n",
+        "        for _, row in df.iterrows():\n",
+        "            nodes, relationships = self.process_entities(row)\n",
+        "\n",
+        "            # Add nodes\n",
+        "            for node in nodes:\n",
+        "                G.add_node(node[\"id\"],\n",
+        "                          type=node[\"type\"],\n",
+        "                          **node[\"properties\"])\n",
+        "\n",
+        "            # Add relationships\n",
+        "            for rel in relationships:\n",
+        "                G.add_edge(rel[\"from\"],\n",
+        "                          rel[\"to\"],\n",
+        "                          relationship=rel[\"type\"],\n",
+        "                          **rel[\"properties\"])\n",
+        "\n",
+        "        return G\n",
+        "\n",
+        "class Neo4jBuilder(GraphBuilder):\n",
+        "    def __init__(self, uri, user, password):\n",
+        "        self.driver = GraphDatabase.driver(uri, auth=(user, password))\n",
+        "        self.logger = logging.getLogger(__name__)\n",
+        "\n",
+        "    def close(self):\n",
+        "        self.driver.close()\n",
+        "\n",
+        "    def build_graph(self, df):\n",
+        "        with self.driver.session() as session:\n",
+        "            for _, row in df.iterrows():\n",
+        "                nodes, relationships = self.process_entities(row)\n",
+        "\n",
+        "                # Create nodes and relationships in Neo4j\n",
+        "                try:\n",
+        "                    session.execute_write(self._create_graph_elements,\n",
+        "                                        nodes, relationships)\n",
+        "                except Exception as e:\n",
+        "                    self.logger.error(f\"Error processing row {row['GKGRECORDID']}: {str(e)}\")\n",
+        "\n",
+        "    def _create_graph_elements(self, tx, nodes, relationships):\n",
+        "        # Create nodes\n",
+        "        for node in nodes:\n",
+        "            query = f\"\"\"\n",
+        "            MERGE (n:{node['type']} {{id: $id}})\n",
+        "            SET n += $properties\n",
+        "            \"\"\"\n",
+        "            tx.run(query, id=node[\"id\"], properties=node[\"properties\"])\n",
+        "\n",
+        "        # Create relationships\n",
+        "        for rel in relationships:\n",
+        "            query = f\"\"\"\n",
+        "            MATCH (a {{id: $from_id}})\n",
+        "            MATCH (b {{id: $to_id}})\n",
+        "            MERGE (a)-[r:{rel['type']}]->(b)\n",
+        "            SET r += $properties\n",
+        "            \"\"\"\n",
+        "            tx.run(query,\n",
+        "                  from_id=rel[\"from\"],\n",
+        "                  to_id=rel[\"to\"],\n",
+        "                  properties=rel[\"properties\"])"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "ghbLZNLe23x1"
+      },
+      "outputs": [],
+      "source": [
+        "if __name__ == \"__main__\":\n",
+        "    # Get data once\n",
+        "    df = get_gdelt_data(limit=25)  # Get 25 records\n",
+        "\n",
+        "    # Build NetworkX graph\n",
+        "    nx_builder = NetworkXBuilder()\n",
+        "    G = nx_builder.build_graph(df)\n",
+        "\n",
+        "    # Print graph information\n",
+        "    print(f\"NetworkX Graph Summary:\")\n",
+        "    print(f\"Nodes: {G.number_of_nodes()}\")\n",
+        "    print(f\"Edges: {G.number_of_edges()}\")\n",
+        "\n",
+        "    # Print node types distribution\n",
+        "    node_types = {}\n",
+        "    for _, attr in G.nodes(data=True):\n",
+        "        node_type = attr.get('type', 'unknown')\n",
+        "        node_types[node_type] = node_types.get(node_type, 0) + 1\n",
+        "\n",
+        "    print(\"\\nNode types distribution:\")\n",
+        "    for ntype, count in node_types.items():\n",
+        "        print(f\"{ntype}: {count}\")\n",
+        "\n",
+        "    # Build Neo4j graph\n",
+        "    neo4j_builder = Neo4jBuilder(URI, USER, PASSWORD)\n",
+        "    try:\n",
+        "        neo4j_builder.build_graph(df)\n",
+        "    finally:\n",
+        "        neo4j_builder.close()"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mkJKz_soTsAY"
+      },
+      "outputs": [],
+      "source": [
+        "# run cypher query for validation\n",
+        "\n",
+        "from neo4j import GraphDatabase\n",
+        "\n",
+        "class Neo4jQuery:\n",
+        "    def __init__(self, uri, user, password):\n",
+        "        self.driver = GraphDatabase.driver(uri, auth=(user, password))\n",
+        "        self.logger = logging.getLogger(__name__)\n",
+        "\n",
+        "    def close(self):\n",
+        "        self.driver.close()\n",
+        "\n",
+        "    def run_query(self, query):\n",
+        "        with self.driver.session() as session:\n",
+        "            result = session.run(query)\n",
+        "            return result.data()\n",
+        "\n",
+        "query_1 = \"\"\"\n",
+        "// Count nodes by type\n",
+        "MATCH (n)\n",
+        "RETURN labels(n) as type, count(*) as count\n",
+        "ORDER BY count DESC;\n",
+        "\"\"\"\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "mrlWADO93ize"
+      },
+      "outputs": [],
+      "source": [
+        "def visualize_graph(G, output_file='gdelt_network.html'):\n",
+        "    \"\"\"Visualize NetworkX graph using Pyvis\"\"\"\n",
+        "    # Create Pyvis network\n",
+        "    net = Network(notebook=True,\n",
+        "                  height='750px',\n",
+        "                  width='100%',\n",
+        "                  bgcolor='#ffffff',\n",
+        "                  font_color='#000000')\n",
+        "\n",
+        "    # Configure physics\n",
+        "    net.force_atlas_2based(gravity=-50,\n",
+        "                          central_gravity=0.01,\n",
+        "                          spring_length=100,\n",
+        "                          spring_strength=0.08,\n",
+        "                          damping=0.4,\n",
+        "                          overlap=0)\n",
+        "\n",
+        "    # Color mapping for node types\n",
+        "    color_map = {\n",
+        "        'event': '#1f77b4',     # Blue\n",
+        "        'person': '#00ff00',    # Green\n",
+        "        'organization': '#ffa500', # Orange\n",
+        "        'location': '#ff0000',  # Red\n",
+        "        'theme': '#800080'      # Purple\n",
+        "    }\n",
+        "\n",
+        "    # Add nodes\n",
+        "    for node, attr in G.nodes(data=True):\n",
+        "        node_type = attr.get('type', 'unknown')\n",
+        "        title = f\"Type: {node_type}\\n\"\n",
+        "        for k, v in attr.items():\n",
+        "            if k != 'type':\n",
+        "                title += f\"{k}: {v}\\n\"\n",
+        "\n",
+        "        net.add_node(node,\n",
+        "                    title=title,\n",
+        "                    label=str(node)[:20] + '...' if len(str(node)) > 20 else str(node),\n",
+        "                    color=color_map.get(node_type, '#gray'),\n",
+        "                    size=20 if node_type == 'event' else 15)\n",
+        "\n",
+        "    # Add edges\n",
+        "    for source, target, attr in G.edges(data=True):\n",
+        "        net.add_edge(source,\n",
+        "                    target,\n",
+        "                    title=f\"{attr.get('relationship', '')}\\nDate: {attr.get('created_at', '')}\",\n",
+        "                    color='#666666')\n",
+        "\n",
+        "    # Save visualization\n",
+        "    net.show(output_file)\n",
+        "    return f\"Graph visualization saved to {output_file}\"\n",
+        "\n",
+        "# Usage example:\n",
+        "if __name__ == \"__main__\":\n",
+        "    visualize_graph(G)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "RqFRO1atnIIT"
+      },
+      "outputs": [],
+      "source": [
+        "!pip show duckdb"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "95ML8u0LnKif"
+      },
+      "outputs": [],
+      "source": []
+    }
+  ],
+  "metadata": {
+    "colab": {
+      "provenance": []
+    },
+    "kernelspec": {
+      "display_name": "Python 3",
+      "name": "python3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

solution_component_notes/gdelt_prefect_extract_to_hf_ds.py

@@ -0,0 +1,303 @@
+import os
+import asyncio
+from prefect import flow, task, get_run_logger
+from prefect.tasks import task_input_hash
+from prefect.blocks.system import Secret, JSON
+from prefect.task_runners import ConcurrentTaskRunner
+from prefect.concurrency.sync import concurrency
+from pathlib import Path
+import datetime
+from datetime import timedelta
+import pandas as pd
+from tqdm import tqdm
+from huggingface_hub import HfApi, hf_hub_url, list_datasets
+import requests
+import zipfile
+from typing import List, Dict, Optional
+
+# --- Constants ---
+# Set a global concurrency limit for Hugging Face uploads
+REPO_ID = "dwb2023/gdelt-gkg-march2020-v2"
+
+BASE_URL = "http://data.gdeltproject.org/gdeltv2"
+
+# Complete Column List
+GKG_COLUMNS = [
+    'GKGRECORDID',                 # Unique identifier
+    'DATE',                        # Publication date
+    'SourceCollectionIdentifier',  # Source type
+    'SourceCommonName',            # Source name
+    'DocumentIdentifier',          # Document URL/ID
+    'V1Counts',                    # Counts of various types
+    'V2.1Counts',                  # Enhanced counts with positions
+    'V1Themes',                    # Theme tags
+    'V2EnhancedThemes',           # Themes with positions
+    'V1Locations',                 # Location mentions
+    'V2EnhancedLocations',        # Locations with positions
+    'V1Persons',                   # Person names
+    'V2EnhancedPersons',          # Persons with positions
+    'V1Organizations',             # Organization names
+    'V2EnhancedOrganizations',     # Organizations with positions
+    'V1.5Tone',                    # Emotional dimensions
+    'V2.1EnhancedDates',          # Date mentions
+    'V2GCAM',                      # Global Content Analysis Measures
+    'V2.1SharingImage',            # Publisher selected image
+    'V2.1RelatedImages',          # Article images
+    'V2.1SocialImageEmbeds',      # Social media images
+    'V2.1SocialVideoEmbeds',      # Social media videos
+    'V2.1Quotations',             # Quote extractions
+    'V2.1AllNames',               # Named entities
+    'V2.1Amounts',                # Numeric amounts
+    'V2.1TranslationInfo',        # Translation metadata
+    'V2ExtrasXML'                 # Additional XML data
+]
+
+# Priority Columns
+PRIORITY_COLUMNS = [
+    'GKGRECORDID',                # Unique identifier
+    'DATE',                       # Publication date
+    'SourceCollectionIdentifier', # Source type
+    'SourceCommonName',           # Source name
+    'DocumentIdentifier',         # Document URL/ID
+    'V1Counts',                   # Numeric mentions
+    'V2.1Counts',                # Enhanced counts
+    'V1Themes',                   # Theme tags
+    'V2EnhancedThemes',          # Enhanced themes
+    'V1Locations',                # Geographic data
+    'V2EnhancedLocations',       # Enhanced locations
+    'V1Persons',                  # Person mentions
+    'V2EnhancedPersons',         # Enhanced persons
+    'V1Organizations',            # Organization mentions
+    'V2EnhancedOrganizations',   # Enhanced organizations
+    'V1.5Tone',                  # Sentiment scores
+    'V2.1EnhancedDates',        # Date mentions
+    'V2GCAM',                    # Enhanced sentiment
+    'V2.1Quotations',           # Direct quotes
+    'V2.1AllNames',             # All named entities
+    'V2.1Amounts'               # Numeric data
+]
+
+# --- Tasks ---
+
+@task(retries=3, retry_delay_seconds=30, log_prints=True)
+def setup_directories(base_path: Path) -> dict:
+    """Create processing directories."""
+    logger = get_run_logger()
+    try:
+        raw_dir = base_path / "gdelt_raw"
+        processed_dir = base_path / "gdelt_processed"
+        raw_dir.mkdir(parents=True, exist_ok=True)
+        processed_dir.mkdir(parents=True, exist_ok=True)
+        logger.info("Directories created successfully")
+        return {"raw": raw_dir, "processed": processed_dir}
+    except Exception as e:
+        logger.error(f"Directory creation failed: {str(e)}")
+        raise
+
+@task(retries=2, log_prints=True)
+def generate_gdelt_urls(start_date: datetime.datetime, end_date: datetime.datetime) -> Dict[datetime.date, List[str]]:
+    """
+    Generate a dictionary keyed by date. Each value is a list of URLs (one per 15-minute interval).
+    """
+    logger = get_run_logger()
+    url_groups = {}
+    try:
+        current_date = start_date.date()
+        while current_date <= end_date.date():
+            urls = [
+                f"{BASE_URL}/{current_date.strftime('%Y%m%d')}{hour:02}{minute:02}00.gkg.csv.zip"
+                for hour in range(24)
+                for minute in [0, 15, 30, 45]
+            ]
+            url_groups[current_date] = urls
+            current_date += timedelta(days=1)
+        logger.info(f"Generated URL groups for dates: {list(url_groups.keys())}")
+        return url_groups
+    except Exception as e:
+        logger.error(f"URL generation failed: {str(e)}")
+        raise
+
+@task(retries=3, retry_delay_seconds=30, log_prints=True)
+def download_file(url: str, raw_dir: Path) -> Path:
+    """Download a single CSV (zip) file from the given URL."""
+    logger = get_run_logger()
+    try:
+        response = requests.get(url, timeout=10)
+        response.raise_for_status()
+        filename = Path(url).name
+        zip_path = raw_dir / filename
+        with zip_path.open('wb') as f:
+            f.write(response.content)
+        logger.info(f"Downloaded {filename}")
+
+        # Optionally, extract the CSV from the ZIP archive.
+        with zipfile.ZipFile(zip_path, 'r') as z:
+            # Assuming the zip contains one CSV file.
+            csv_names = z.namelist()
+            if csv_names:
+                extracted_csv = raw_dir / csv_names[0]
+                z.extractall(path=raw_dir)
+                logger.info(f"Extracted {csv_names[0]}")
+                return extracted_csv
+            else:
+                raise ValueError("Zip file is empty.")
+    except Exception as e:
+        logger.error(f"Error downloading {url}: {str(e)}")
+        raise
+
+@task(retries=2, log_prints=True)
+def convert_and_filter_combined(csv_paths: List[Path], processed_dir: Path, date: datetime.date) -> Path:
+    """
+    Combine multiple CSV files (for one day) into a single DataFrame,
+    filter to only the required columns, optimize data types,
+    and write out as a single Parquet file.
+    """
+    logger = get_run_logger()
+    try:
+        dfs = []
+        for csv_path in csv_paths:
+            df = pd.read_csv(
+                csv_path,
+                sep='\t',
+                names=GKG_COLUMNS,
+                dtype='string',
+                quoting=3,
+                na_values=[''],
+                encoding='utf-8',
+                encoding_errors='replace'
+            )
+            dfs.append(df)
+        combined_df = pd.concat(dfs, ignore_index=True)
+        filtered_df = combined_df[PRIORITY_COLUMNS].copy()
+        # Convert the date field to datetime; adjust the format if necessary.
+        if 'V2.1DATE' in filtered_df.columns:
+            filtered_df['V2.1DATE'] = pd.to_datetime(
+                filtered_df['V2.1DATE'], format='%Y%m%d%H%M%S', errors='coerce'
+            )
+        output_filename = f"gdelt_gkg_{date.strftime('%Y%m%d')}.parquet"
+        output_path = processed_dir / output_filename
+        filtered_df.to_parquet(output_path, engine='pyarrow', compression='snappy', index=False)
+        logger.info(f"Converted and filtered data for {date} into {output_filename}")
+        return output_path
+    except Exception as e:
+        logger.error(f"Error processing CSVs for {date}: {str(e)}")
+        raise
+
+@task(retries=3, retry_delay_seconds=30, log_prints=True)
+def upload_to_hf(file_path: Path, token: str) -> bool:
+    """Upload task with global concurrency limit."""
+    logger = get_run_logger()
+    try:
+        with concurrency("hf_uploads", occupy=1):
+            # Enable the optimized HF Transfer backend.
+            os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
+
+            api = HfApi()
+            api.upload_file(
+                path_or_fileobj=str(file_path),
+                path_in_repo=file_path.name,
+                repo_id=REPO_ID,
+                repo_type="dataset",
+                token=token,
+            )
+            logger.info(f"Uploaded {file_path.name}")
+        return True
+    except Exception as e:
+        logger.error(f"Upload failed for {file_path.name}: {str(e)}")
+        raise
+
+@task(retries=3, retry_delay_seconds=120, log_prints=True)
+def create_hf_repo(token: str) -> bool:
+    """
+    Validate that the Hugging Face dataset repository exists; create it if not.
+    """
+    logger = get_run_logger()
+    try:
+        api = HfApi()
+        datasets = [ds.id for ds in list_datasets(token=token)]
+        if REPO_ID in datasets:
+            logger.info(f"Dataset repository '{REPO_ID}' already exists.")
+            return True
+        # Create the repository if it doesn't exist.
+        api.create_repo(repo_id=REPO_ID, repo_type="dataset", token=token, private=False)
+        logger.info(f"Successfully created dataset repository: {REPO_ID}")
+        return True
+    except Exception as e:
+        logger.error(f"Failed to create or validate dataset repo '{REPO_ID}': {str(e)}")
+        raise RuntimeError(f"Repository validation/creation failed for '{REPO_ID}'") from e
+
+@flow(name="Process Single Day", log_prints=True)
+def process_single_day(
+    date: datetime.date, urls: List[str], directories: dict, hf_token: str
+) -> bool:
+    """
+    Process one day's data by:
+      1. Downloading all CSV files concurrently.
+      2. Merging, filtering, and optimizing the CSVs.
+      3. Writing out a single daily Parquet file.
+      4. Uploading the file to the Hugging Face Hub.
+    """
+    logger = get_run_logger()
+    try:
+        # Download and process data (unlimited concurrency)
+        csv_paths = [download_file(url, directories["raw"]) for url in urls]
+        daily_parquet = convert_and_filter_combined(csv_paths, directories["processed"], date)
+        
+        # Upload with global concurrency limit
+        upload_to_hf(daily_parquet, hf_token)  # <-- Throttled to 2 concurrent
+    
+        logger.info(f"Completed {date}")
+        return True
+    except Exception as e:
+        logger.error(f"Day {date} failed: {str(e)}")
+        raise
+
+@flow(
+    name="Process Date Range",
+    task_runner=ConcurrentTaskRunner(),  # Parallel subflows
+    log_prints=True
+)
+def process_date_range(base_path: Path = Path("data")):
+    """
+    Main ETL flow:
+      1. Load parameters and credentials.
+      2. Validate (or create) the Hugging Face repository.
+      3. Setup directories.
+      4. Generate URL groups by date.
+      5. Process each day concurrently.
+    """
+    logger = get_run_logger()
+
+    # Load parameters from a JSON block.
+    json_block = JSON.load("gdelt-etl-parameters")
+    params = json_block.value
+    start_date = datetime.datetime.fromisoformat(params.get("start_date", "2020-03-16T00:00:00"))
+    end_date = datetime.datetime.fromisoformat(params.get("end_date", "2020-03-22T00:00:00"))
+
+    # Load the Hugging Face token from a Secret block.
+    secret_block = Secret.load("huggingface-token")
+    hf_token = secret_block.get()
+
+    # Validate or create the repository.
+    create_hf_repo(hf_token)
+
+    directories = setup_directories(base_path)
+    url_groups = generate_gdelt_urls(start_date, end_date)
+    
+    # Process days concurrently (subflows)
+    futures = [process_single_day(date, urls, directories, hf_token) 
+              for date, urls in url_groups.items()]
+    
+    # Wait for completion (optional error handling)
+    for future in futures:
+        try:
+            future.result()
+        except Exception as e:
+            logger.error(f"Failed day: {str(e)}")
+
+# --- Entry Point ---
+if __name__ == "__main__":
+    process_date_range.serve(
+        name="gdelt-etl-production-v2",
+        tags=["gdelt", "etl", "production"],
+    )

solution_component_notes/hf_gdelt_dataset_2020_covid.md

@@ -0,0 +1,166 @@
+---
+license: cc-by-4.0
+tags:
+- text
+- news
+- global
+- knowledge-graph
+- geopolitics
+dataset_info:
+  features:
+  - name: GKGRECORDID
+    dtype: string
+  - name: DATE
+    dtype: string
+  - name: SourceCollectionIdentifier
+    dtype: string
+  - name: SourceCommonName
+    dtype: string
+  - name: DocumentIdentifier
+    dtype: string
+  - name: V1Counts
+    dtype: string
+  - name: V2.1Counts
+    dtype: string
+  - name: V1Themes
+    dtype: string
+  - name: V2EnhancedThemes
+    dtype: string
+  - name: V1Locations
+    dtype: string
+  - name: V2EnhancedLocations
+    dtype: string
+  - name: V1Persons
+    dtype: string
+  - name: V2EnhancedPersons
+    dtype: string
+  - name: V1Organizations
+    dtype: string
+  - name: V2EnhancedOrganizations
+    dtype: string
+  - name: V1.5Tone
+    dtype: string
+  - name: V2GCAM
+    dtype: string
+  - name: V2.1EnhancedDates
+    dtype: string
+  - name: V2.1Quotations
+    dtype: string
+  - name: V2.1AllNames
+    dtype: string
+  - name: V2.1Amounts
+    dtype: string
+  - name: tone
+    dtype: float64
+  splits:
+  - name: train
+    num_bytes: 3331097194
+    num_examples: 281215
+  - name: negative_tone
+    num_bytes: 3331097194
+    num_examples: 281215
+  download_size: 2229048020
+  dataset_size: 6662194388
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: negative_tone
+    path: data/negative_tone-*
+---
+
+# Dataset Card for dwb2023/gdelt-gkg-march2020-v2
+
+## Dataset Details
+
+### Dataset Description
+
+This dataset contains GDELT Global Knowledge Graph (GKG) data covering March 10-22, 2020, during the early phase of the COVID-19 pandemic. It captures global event interactions, actor relationships, and contextual narratives to support temporal, spatial, and thematic analysis.
+
+- **Curated by:** dwb2023
+
+### Dataset Sources
+
+- **Repository:** [http://data.gdeltproject.org/gdeltv2](http://data.gdeltproject.org/gdeltv2)
+- **GKG Documentation:** [GDELT 2.0 Overview](https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/), [GDELT GKG Codebook](http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook-V2.1.pdf)
+
+## Uses
+
+### Direct Use
+
+This dataset is suitable for:
+
+- Temporal analysis of global events
+- Relationship mapping of key actors in supply chain and logistics
+- Sentiment and thematic analysis of COVID-19 pandemic narratives
+
+### Out-of-Scope Use
+
+- Not designed for real-time monitoring due to its historic and static nature
+- Not intended for medical diagnosis or predictive health modeling
+
+## Dataset Structure
+
+### Features and Relationships
+
+- this dataset focuses on a subset of features from the source GDELT dataset.
+
+| Name | Type | Aspect | Description |
+|------|------|---------|-------------|
+| DATE | string | Metadata | Publication date of the article/document |
+| SourceCollectionIdentifier | string | Metadata | Unique identifier for the source collection |
+| SourceCommonName | string | Metadata | Common/display name of the source |
+| DocumentIdentifier | string | Metadata | Unique URL/identifier of the document |
+| V1Counts | string | Metrics | Original count mentions of numeric values |
+| V2.1Counts | string | Metrics | Enhanced numeric pattern extraction |
+| V1Themes | string | Classification | Original thematic categorization |
+| V2EnhancedThemes | string | Classification | Expanded theme taxonomy and classification |
+| V1Locations | string | Entities | Original geographic mentions |
+| V2EnhancedLocations | string | Entities | Enhanced location extraction with coordinates |
+| V1Persons | string | Entities | Original person name mentions |
+| V2EnhancedPersons | string | Entities | Enhanced person name extraction |
+| V1Organizations | string | Entities | Original organization mentions |
+| V2EnhancedOrganizations | string | Entities | Enhanced organization name extraction |
+| V1.5Tone | string | Sentiment | Original emotional tone scoring |
+| V2GCAM | string | Sentiment | Global Content Analysis Measures |
+| V2.1EnhancedDates | string | Temporal | Temporal reference extraction |
+| V2.1Quotations | string | Content | Direct quote extraction |
+| V2.1AllNames | string | Entities | Comprehensive named entity extraction |
+| V2.1Amounts | string | Metrics | Quantity and measurement extraction |
+
+### Aspects Overview:
+- **Metadata**: Core document information
+- **Metrics**: Numerical measurements and counts
+- **Classification**: Categorical and thematic analysis
+- **Entities**: Named entity recognition (locations, persons, organizations)
+- **Sentiment**: Emotional and tone analysis
+- **Temporal**: Time-related information
+- **Content**: Direct content extraction
+
+## Dataset Creation
+
+### Curation Rationale
+This dataset was curated to capture the rapidly evolving global narrative during the early phase of the COVID-19 pandemic, focusing specifically on March 10–22, 2020. By zeroing in on this critical period, it offers a granular perspective on how geopolitical events, actor relationships, and thematic discussions shifted amid the escalating pandemic. The enhanced GKG features further enable advanced entity, sentiment, and thematic analysis, making it a valuable resource for studying the socio-political and economic impacts of COVID-19 during a pivotal point in global history.
+
+### Curation Approach
+A targeted subset of GDELT’s columns was selected to streamline analysis on key entities (locations, persons, organizations), thematic tags, and sentiment scores—core components of many knowledge-graph and text analytics workflows. This approach balances comprehensive coverage with manageable data size and performance. The ETL pipeline used to produce these transformations is documented here:
+[https://gist.github.com/donbr/e2af2bbe441f90b8664539a25957a6c0](https://gist.github.com/donbr/e2af2bbe441f90b8664539a25957a6c0).
+
+## Citation
+
+When using this dataset, please cite both the dataset and original GDELT project:
+
+```bibtex
+@misc{gdelt-gkg-march2020,
+    title = {GDELT Global Knowledge Graph March 2020 Dataset},
+    author = {dwb2023},
+    year = {2025},
+    publisher = {Hugging Face},
+    url = {https://huggingface.co/datasets/dwb2023/gdelt-gkg-march2020-v2}
+}
+```
+
+## Dataset Card Contact
+
+For questions and comments about this dataset card, please contact dwb2023 through the Hugging Face platform.

solution_component_notes/hf_gdelt_dataset_2025_february.md

@@ -0,0 +1,149 @@
+---
+license: cc-by-4.0
+tags:
+- text
+- news
+- global
+- knowledge-graph
+- geopolitics
+dataset_info:
+  features:
+  - name: GKGRECORDID
+    dtype: string
+  - name: DATE
+    dtype: string
+  - name: SourceCollectionIdentifier
+    dtype: string
+  - name: SourceCommonName
+    dtype: string
+  - name: DocumentIdentifier
+    dtype: string
+  - name: V1Counts
+    dtype: string
+  - name: V2.1Counts
+    dtype: string
+  - name: V1Themes
+    dtype: string
+  - name: V2EnhancedThemes
+    dtype: string
+  - name: V1Locations
+    dtype: string
+  - name: V2EnhancedLocations
+    dtype: string
+  - name: V1Persons
+    dtype: string
+  - name: V2EnhancedPersons
+    dtype: string
+  - name: V1Organizations
+    dtype: string
+  - name: V2EnhancedOrganizations
+    dtype: string
+  - name: V1.5Tone
+    dtype: string
+  - name: V2.1EnhancedDates
+    dtype: string
+  - name: V2GCAM
+    dtype: string
+  - name: V2.1SharingImage
+    dtype: string
+  - name: V2.1Quotations
+    dtype: string
+  - name: V2.1AllNames
+    dtype: string
+  - name: V2.1Amounts
+    dtype: string
+---
+
+# Dataset Card for dwb2023/gdelt-gkg-2025-v2
+
+## Dataset Details
+
+### Dataset Description
+
+This dataset contains GDELT Global Knowledge Graph (GKG) data covering February 2025. It captures global event interactions, actor relationships, and contextual narratives to support temporal, spatial, and thematic analysis.
+
+- **Curated by:** dwb2023
+
+### Dataset Sources
+
+- **Repository:** [http://data.gdeltproject.org/gdeltv2](http://data.gdeltproject.org/gdeltv2)
+- **GKG Documentation:** [GDELT 2.0 Overview](https://blog.gdeltproject.org/gdelt-2-0-our-global-world-in-realtime/), [GDELT GKG Codebook](http://data.gdeltproject.org/documentation/GDELT-Global_Knowledge_Graph_Codebook-V2.1.pdf)
+
+## Uses
+
+### Direct Use
+
+This dataset is suitable for:
+
+- Temporal analysis of global events
+
+### Out-of-Scope Use
+
+- Not designed for real-time monitoring due to its historic and static nature
+- Not intended for medical diagnosis or predictive health modeling
+
+## Dataset Structure
+
+### Features and Relationships
+
+- this dataset focuses on a subset of features from the source GDELT dataset.
+
+| Name | Type | Aspect | Description |
+|------|------|---------|-------------|
+| DATE | string | Metadata | Publication date of the article/document |
+| SourceCollectionIdentifier | string | Metadata | Unique identifier for the source collection |
+| SourceCommonName | string | Metadata | Common/display name of the source |
+| DocumentIdentifier | string | Metadata | Unique URL/identifier of the document |
+| V1Counts | string | Metrics | Original count mentions of numeric values |
+| V2.1Counts | string | Metrics | Enhanced numeric pattern extraction |
+| V1Themes | string | Classification | Original thematic categorization |
+| V2EnhancedThemes | string | Classification | Expanded theme taxonomy and classification |
+| V1Locations | string | Entities | Original geographic mentions |
+| V2EnhancedLocations | string | Entities | Enhanced location extraction with coordinates |
+| V1Persons | string | Entities | Original person name mentions |
+| V2EnhancedPersons | string | Entities | Enhanced person name extraction |
+| V1Organizations | string | Entities | Original organization mentions |
+| V2EnhancedOrganizations | string | Entities | Enhanced organization name extraction |
+| V1.5Tone | string | Sentiment | Original emotional tone scoring |
+| V2.1EnhancedDates | string | Temporal | Temporal reference extraction |
+| V2GCAM | string | Sentiment | Global Content Analysis Measures |
+| V2.1SharingImage | string | Content | URL of document image |
+| V2.1Quotations | string | Content | Direct quote extraction |
+| V2.1AllNames | string | Entities | Comprehensive named entity extraction |
+| V2.1Amounts | string | Metrics | Quantity and measurement extraction |
+
+### Aspects Overview:
+- **Metadata**: Core document information
+- **Metrics**: Numerical measurements and counts
+- **Classification**: Categorical and thematic analysis
+- **Entities**: Named entity recognition (locations, persons, organizations)
+- **Sentiment**: Emotional and tone analysis
+- **Temporal**: Time-related information
+- **Content**: Direct content extraction
+
+## Dataset Creation
+
+### Curation Rationale
+This dataset was curated to capture the rapidly evolving global narrative during February 2025. By zeroing in on this critical period, it offers a granular perspective on how geopolitical events, actor relationships, and thematic discussions shifted amid the escalating pandemic. The enhanced GKG features further enable advanced entity, sentiment, and thematic analysis, making it a valuable resource for studying the socio-political and economic impacts of emergent LLM capabilities.
+
+### Curation Approach
+A targeted subset of GDELT’s columns was selected to streamline analysis on key entities (locations, persons, organizations), thematic tags, and sentiment scores—core components of many knowledge-graph and text analytics workflows. This approach balances comprehensive coverage with manageable data size and performance. The ETL pipeline used to produce these transformations is documented here:
+[https://gist.github.com/donbr/5293468436a1a39bd2d9f4959cbd4923](https://gist.github.com/donbr/5293468436a1a39bd2d9f4959cbd4923).
+
+## Citation
+
+When using this dataset, please cite both the dataset and original GDELT project:
+
+```bibtex
+@misc{gdelt-gkg-2025-v2,
+    title = {GDELT Global Knowledge Graph 2025 Dataset},
+    author = {dwb2023},
+    year = {2025},
+    publisher = {Hugging Face},
+    url = {https://huggingface.co/datasets/dwb2023/gdelt-gkg-2025-v2}
+}
+```
+
+## Dataset Card Contact
+
+For questions and comments about this dataset card, please contact dwb2023 through the Hugging Face platform.