Move load-config to graphrag-common.

Author: dworthen

docs/examples_notebooks/api_overview.ipynb

@@ -16,7 +16,7 @@
    "source": [
     "## API Overview\n",
     "\n",
-    "This notebook provides a demonstration of how to interact with graphrag as a library using the API as opposed to the CLI. Note that graphrag's CLI actually connects to the library through this API for all operations. "
+    "This notebook provides a demonstration of how to interact with graphrag as a library using the API as opposed to the CLI. Note that graphrag's CLI actually connects to the library through this API for all operations.\n"
    ]
   },
   {
@@ -28,11 +28,10 @@
     "from pathlib import Path\n",
     "from pprint import pprint\n",
     "\n",
+    "import graphrag.api as api\n",
     "import pandas as pd\n",
-    "from graphrag.config.load_config import load_config\n",
     "from graphrag.index.typing.pipeline_run_result import PipelineRunResult\n",
-    "\n",
-    "import graphrag.api as api"
+    "from graphrag_common.config import load_config"
    ]
   },
   {
@@ -49,16 +48,17 @@
    "metadata": {},
    "source": [
     "## Prerequisite\n",
+    "\n",
     "As a prerequisite to all API operations, a `GraphRagConfig` object is required. It is the primary means to control the behavior of graphrag and can be instantiated from a `settings.yaml` configuration file.\n",
     "\n",
-    "Please refer to the [CLI docs](https://microsoft.github.io/graphrag/cli/#init) for more detailed information on how to generate the `settings.yaml` file."
+    "Please refer to the [CLI docs](https://microsoft.github.io/graphrag/cli/#init) for more detailed information on how to generate the `settings.yaml` file.\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Generate a `GraphRagConfig` object"
+    "### Generate a `GraphRagConfig` object\n"
    ]
   },
   {
@@ -78,14 +78,14 @@
    "source": [
     "## Indexing API\n",
     "\n",
-    "*Indexing* is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats."
+    "_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Build an index"
+    "## Build an index\n"
    ]
   },
   {
@@ -108,7 +108,7 @@
    "source": [
     "## Query an index\n",
     "\n",
-    "To query an index, several index files must first be read into memory and passed to the query API. "
+    "To query an index, several index files must first be read into memory and passed to the query API.\n"
    ]
   },
   {
@@ -139,7 +139,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response."
+    "The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
    ]
   },
   {
@@ -155,7 +155,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model)."
+    "Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
    ]
   },
   {

docs/examples_notebooks/input_documents.ipynb

@@ -18,7 +18,7 @@
     "\n",
     "Newer versions of GraphRAG let you submit a dataframe directly instead of running through the input processing step. This notebook demonstrates with regular or update runs.\n",
     "\n",
-    "If performing an update, the assumption is that your dataframe contains only the new documents to add to the index."
+    "If performing an update, the assumption is that your dataframe contains only the new documents to add to the index.\n"
    ]
   },
   {
@@ -30,11 +30,10 @@
     "from pathlib import Path\n",
     "from pprint import pprint\n",
     "\n",
+    "import graphrag.api as api\n",
     "import pandas as pd\n",
-    "from graphrag.config.load_config import load_config\n",
     "from graphrag.index.typing.pipeline_run_result import PipelineRunResult\n",
-    "\n",
-    "import graphrag.api as api"
+    "from graphrag_common.config import load_config"
    ]
   },
   {
@@ -55,7 +54,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Generate a `GraphRagConfig` object"
+    "### Generate a `GraphRagConfig` object\n"
    ]
   },
   {
@@ -73,14 +72,14 @@
    "source": [
     "## Indexing API\n",
     "\n",
-    "*Indexing* is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats."
+    "_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Build an index"
+    "## Build an index\n"
    ]
   },
   {
@@ -110,7 +109,7 @@
    "source": [
     "## Query an index\n",
     "\n",
-    "To query an index, several index files must first be read into memory and passed to the query API. "
+    "To query an index, several index files must first be read into memory and passed to the query API.\n"
    ]
   },
   {
@@ -141,7 +140,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response."
+    "The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
    ]
   },
   {
@@ -157,7 +156,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model)."
+    "Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
    ]
   },
   {

packages/graphrag-common/README.md

@@ -48,4 +48,65 @@ single2 = factory.create("some_other_strategy", {"value": "ignored"})
 assert single1 is single2
 assert single1.get_value() == "singleton"
 assert single2.get_value() == "singleton"
+```
+
+## Config module
+
+```python
+from pydantic import BaseModel, Field
+from graphrag_common.config import load_config
+
+from pathlib import Path
+
+class Logging(BaseModel):
+    """Test nested model."""
+
+    directory: str = Field(default="output/logs")
+    filename: str = Field(default="logs.txt")
+
+class Config(BaseModel):
+    """Test configuration model."""
+
+    name: str = Field(description="Name field.")
+    logging: Logging = Field(description="Nested model field.")
+
+# Basic
+# By default, ${} env variables in config file will be parsed and replaced
+# can disable by setting parse_env_vars=False
+config = load_config(Config, "path/to/config.[yaml|yml|json]")
+
+# with .env file
+config = load_config(
+    config_initializer=Config,
+    config_path="config.yaml",
+    dot_env_path=".env"
+)
+
+# With overrides - provided values override whats in the config file
+# Only overrides what is specified - recursively merges settings.
+config = load_config(
+    config_initializer=Config,
+    config_path="config.yaml",
+    overrides={
+        "name": "some name"
+        "logging": {
+            "filename": "my_logs.txt"
+        }
+    },
+)
+
+# Set the working directory to the directory of the config file
+config = load_config(
+    config_initializer=Config,
+    config_path="some/path/to/config.yaml",
+    set_cwd=True
+)
+
+# now cwd == some/path/to
+assert Path.cwd() == "some/path/to"
+
+# And now throughout the codebase resolving relative paths in config
+# will resolve relative to the config directory
+Path(config.logging.directory) == "some/path/to/output/logging"
+
 ```

packages/graphrag-common/graphrag_common/config/__init__.py

@@ -0,0 +1,8 @@
+# Copyright (c) 2024 Microsoft Corporation.
+# Licensed under the MIT License
+
+"""The GraphRAG config module."""
+
+from graphrag_common.config.load_config import ConfigParsingError, load_config
+
+__all__ = ["ConfigParsingError", "load_config"]

packages/graphrag-config/graphrag_config/load_config.py renamed to packages/graphrag-common/graphrag_common/config/load_config.py

@@ -3,19 +3,54 @@
 
 """Load configuration."""
 
+import json
 import os
 from collections.abc import Callable
 from pathlib import Path
 from string import Template
 from typing import Any, TypeVar
 
+import yaml
 from dotenv import load_dotenv
 
-from graphrag_config.parsers import ConfigParsingError, get_parser_for_file
-
 T = TypeVar("T", covariant=True)
 
 
+class ConfigParsingError(ValueError):
+    """Configuration Parsing Error."""
+
+    def __init__(self, msg: str) -> None:
+        """Initialize the ConfigParsingError."""
+        super().__init__(msg)
+
+
+def _parse_json(data: str) -> dict[str, Any]:
+    """Parse JSON data."""
+    return json.loads(data)
+
+
+def _parse_yaml(data: str) -> dict[str, Any]:
+    """Parse YAML data."""
+    return yaml.safe_load(data)
+
+
+def _get_parser_for_file(file_path: str | Path) -> Callable[[str], dict[str, Any]]:
+    """Get the parser for the given file path."""
+    file_path = Path(file_path).resolve()
+    match file_path.suffix.lower():
+        case ".json":
+            return _parse_json
+        case ".yaml" | ".yml":
+            return _parse_yaml
+        case _:
+            msg = (
+                f"Failed to parse, {file_path}. Unsupported file extension, "
+                + f"{file_path.suffix}. Pass in a custom config_parser argument or "
+                + "use one of the supported file extensions, .json, .yaml, .yml, .toml."
+            )
+            raise ConfigParsingError(msg)
+
+
 def _load_dotenv(env_file_path: Path | str) -> None:
     """Load the .env file if it exists."""
     env_file_path = Path(env_file_path).resolve()
@@ -81,7 +116,7 @@ def load_config(
     config_parser : Callable[[str], dict[str, Any]] | None, optional (default=None)
         function to parse the configuration text, (str) -> dict[str, Any].
         If None, the parser is inferred from the file extension.
-        Supported extensions: .json, .yaml, .yml, .toml.
+        Supported extensions: .json, .yaml, .yml.
     file_encoding : str, optional (default="utf-8")
         File encoding to use when reading the configuration file.
 
@@ -99,7 +134,6 @@ def load_config(
     ConfigParsingError
         - If an environment variable is not found when parsing env variables.
         - If there was a problem merging the overrides with the configuration.
-        - If there was a problem initializing the configuration with the config_initializer.
         - If parser=None and load_config was unable to determine how to parse
         the file based on the file extension.
         - If the parser fails to parse the configuration text.
@@ -123,13 +157,13 @@ def load_config(
         file_contents = _parse_env_variables(file_contents)
 
     if config_parser is None:
-        config_parser = get_parser_for_file(config_path)
+        config_parser = _get_parser_for_file(config_path)
 
     config_data: dict[str, Any] = {}
     try:
         config_data = config_parser(file_contents)
     except Exception as error:
-        msg = f"Failed to parse config_path: {config_path}"
+        msg = f"Failed to parse config_path: {config_path}. Error: {error}"
         raise ConfigParsingError(msg) from error
 
     if overrides is not None: