flexible, and config-driven pipeline for loading workflow data into Neo4j #38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces YAML and Pydantic-based schema definitions for nodes and relationships in the ingestion pipeline, enabling flexible, strongly-typed modeling of workflows, tools, categories, steps, and their interconnections. Facilitates easier schema evolution and validation for ingesting graph-structured data.
Introduces a generic, schema-flexible loader to convert workflow JSON into normalized CSVs and ingest them into Neo4j via a configurable mapping. Enables reusable, idempotent data import of workflows, tools, and relationships without hardcoding schema, supporting future extensibility and simplified dataset integration.
Introduces a new relationship definition to model the association between workflow steps and tools, enabling more granular tracking of tool usage within workflows. This enhances data integrity and supports future analytical capabilities regarding step dependencies.
Streamlines the loader file by deleting an extensive docstring with usage instructions and configuration notes. Aims to reduce redundancy and improve maintainability, assuming documentation is maintained elsewhere.
Introduces a utility for converting CSV-based node and relationship data into Cypher statements suitable for database import. Automates property handling, identifier generation, and supports configurable schemas to streamline graph data loading workflows.
Introduces a script to execute generated Cypher files for batch-loading nodes and edges into Neo4j. Automates the import process by reading Cypher files, splitting statements, and running them sequentially, improving reproducibility and simplifying data ingestion workflows. Adds batch Cypher loader for automated Neo4j imports Introduces a script to automate loading nodes and edges into Neo4j from generated Cypher files. Enhances reproducibility and simplifies the data ingestion workflow by running Cypher statements in order, reducing manual intervention and potential for errors.
aprilyab
pushed a commit
to aprilyab/galaxy-agent-xp-II
that referenced
this pull request
Dec 28, 2025
…pe-and-dynamic-configs flexible, and config-driven pipeline for loading workflow data into Neo4j
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new, flexible, and config-driven pipeline for loading workflow data into Neo4j, designed to support multiple data schemas and use cases. The pipeline includes scripts for converting workflow JSON to CSV, generating Cypher files, and loading data directly or via Cypher into Neo4j. The changes are grouped as follows:
Pipeline Implementation and Data Conversion:
convert_json_to_csv.pyto flatten workflow JSON into multiple CSV files for workflows, workflow files, steps, inputs, outputs, input connections, and tools. This script ensures schema flexibility and supports optional tool extraction.README.mddescribing the pipeline architecture, components, ID strategy, relationships, and usage instructions for the generic loader.README.mdin thedatadirectory.Neo4j Loading and Cypher Generation:
generic_csv_loader.py, a config-driven loader that ingests the generated CSVs into Neo4j using deterministic IDs and merges nodes and relationships based on the configuration.generate_cypher_files.pyto emit Cypher files (nodes.cypherandedges.cypher) from CSVs, using the same configuration and ID strategy as the loader.cypher_batch_loader.py, which executes the generated Cypher files against Neo4j, processing nodes before edges for correct graph construction.