📋
Philanthrolab
  • Philanthrolab Technical Docs
  • SSN Component Library
  • Datalabs
    • Introduction
    • Architecture
    • Schema Dictionary
    • Project Status/Timeline
  • Social Safety Network
    • Introduction
    • Architecture
    • Schema Dictionary
    • Project Status/Timeline
      • V1
      • V2
  • SSN for Organisations
    • Introduction
    • Features and user stories
    • Architecture
    • Schema Dictionary
    • Project Status/Timeline
  • Developer Resources
    • Frontend Project Guide
    • Coding Guide
    • Creating a Neo4j instance on GCP vm
    • Set up local deploy for staging and production envs
    • Install ElasticSearch on GCP
    • ElasticSearch Query
    • ETL Strategy for Neo4j Database: Scraping, Transformation, and Enrichment
    • ETL Checklist
  • SSN Authentication
    • Introduction
    • Architecture
    • Schema
  • SSN Admin Dashboard
    • Introduction
    • Architecture
  • SSN Job Board
    • Introduction
    • Architecture
    • User Stories
    • Schema Dictionary
  • SSN Eligibility criteria AI feature
    • Introduction
    • Working Principles
    • Architecture
    • Schema Dictionary
  • DataBase Repopulation
    • Introduction
    • Proposed Solution
    • DB Details
    • Batch 1
  • LLM INTEGRATION
    • LLM Strategy and Implementation
Powered by GitBook
On this page

Was this helpful?

  1. DataBase Repopulation

Proposed Solution

How can we solve this problems to avoid future data irregularities and errors?

Since the data in both production and staging are not reproducible because of google places API and lack of APOC plugin compatible with those version of the DB, We can create a SandBox db and export useful data gradually or in batches for proper cleaning and restructuring. Test against backend codebase for code breaking in data querying.

  • Data irregularities in both DB : We focus on only the production DB

  • No documentation on data processing for reproducibility

  • Incompatibility of APOC plugin on the DBs version for total data export

Issues
Solutions
Remarks

Data irregularities in both DB

Focus only on production DB

Less data but useful

No Data Processing Documentation

Give brief description on each batch data processing

Increases reproducibilities chances

Incompatibility of APOC plugin on the DBs version for total data export

The new DB must be APOC compatible

Good for total data export in the case of DB migration in future.

PreviousIntroductionNextDB Details

Last updated 2 years ago

Was this helpful?