Skip to main content
Integration tests verify the overall functionality of the application by testing interactions between different components (calculations, rendering of data, etc.) work as expected.

​
Overview

Considering that Maybern is in the business of automating core calculations for managing private funds, data accuracy is paramount. The integration testing suite ensures that data we generate is accurate and consistent across all customers—we should always compute and view data the same way day over day unless an expected code change is made. Integration tests are designed to be flexible so we can layer in more tests as we add more features and customers.

​
Testing Customers

​
Long Lived Test Customers

Previously, all integration tests would create a new customer per test run, either by loading an Excel sheet or JSON template. This gave us a lot of coverage, but did not cover scenarios such as migrations or changes to existing customers. Long Lived Test Customers are customers that have tests run on them on a regular basis, rather than being recreated from scratch for each test run. These customers are automatically set up with delete protection. You can see the list of Long Lived Test Customers in sandbox at /internal/integration-testing/configurations.

​
Job Types

We currently have three types of integration tests. They differ in how the test customer is chosen, but all three run the same set of validation checks using the same set of expected data.

​
V1 (Excel - Legacy)

  • Loads data via shadow into new test customers
  • Runs validation checks on the loaded customers
  • Checks shadow load specific logic along with writes and reads across the app

​
Long Lived Customer Validation

  • Runs validation checks on the long lived customers
  • Primary purpose is to see if a migration has broken something

​
Long Lived Customer Reload and Validate

  • Export long lived customer
  • Reload into fresh test customer
  • Run validation checks on the fresh test customer
  • Validates event export and loading logic along with writes and reads across the app

​
Job Cadences

Different customers run at different cadences throughout the day. We currently have tests that run once, twice, or four times a day. Customer configs are grouped by their cadence type. When we run a specific cadence, we run tests for that cadence and all more frequent ones—so a run that triggers the once-a-day group will also run the twice and four-times-a-day groups. The jobs that run integration tests are defined in Terraform.

​
Test Environments

EnvironmentCadenceGolden Datasets Location
PR TestingEvery push/mergeGitHub Repo
Sandbox4 times/dayS3
Staging1 time/dayS3

​
Validation Types

​
Transaction Integration Tests

Compare how features are computed and persisted into the DB.

​
API Integration Tests

Compare the data surfaced to end users via API.

​
Export Tests

Ensure that a customer can be exported at the end of a run.

​
Workflow

Breaking down a typical integration test run:
  1. Read and validate transactions in parallel by type and customer
  2. Read and validate API data in parallel by type, sample number, and customer
  3. Export template data by customer in parallel (if configured)

​
What “Read and Validate” Means

  • Read: Query the database for specified set of transactions, hit an API, etc. (depending on test type) against the newly provisioned customer
  • Validate: Compare the exported data against a snapshot of that exact same test from a previous integration test
When running a test, we compare the read data against a snapshot from a previous successful run. If the data is different, the test fails. This tells us that an API has changed its contract or numbers are being calculated differently. We can then tie that change back to the specific PR and code change that caused the discrepancy.

​
Debugging

When debugging integration tests, understand the toolkit we have for understanding what went wrong.

​
Individual Test Artifacts

  • Actual data: Export from the current run
  • Expected data: Snapshot from the last successful run
  • Comparison data: Diff between the actual and expected data

​
System-Level Tracking

  • PRs are labeled with any tests that may have been updated (integration-test-api, integration-test-transaction, etc.)
  • We track all git commits that go into a test run
  • We compare test results against previous integration tests to identify new failures, existing failures, or resolved failures
  • We compare enabled feature flags against the last successful run to identify flag changes

​
Models

​
Configuration Models

  • IntegrationTestingConfiguration: Defines the set of customers/templates to run in an integration test

​
Queue Models

  • TestingQueue: Tracks the type of integration test to run (for the cron to pick up)
  • TestingQueueItem: Tracks an individual customer/test to run associated with the test type

​
Report Models

  • TestingReport: Tracks a full integration test
  • IndividualTestingReport: Tracks a single test within an integration test

​
Misc Models

  • TestingReportIssue: Tracks an issue found in an integration test that is long running

​
Local Development

To load customers for testing, run the following commands:

​
Setup Backend Server and Workers

# Terminal 1: Run Django server
DJANGO_ENV=integration_test ./manage.py runserver

# Terminal 2: Run Celery workers
DJANGO_ENV=integration_test ./manage.py run_celery

​
V1 Test

# Terminal 3: Run shadow integration testing
SHADOW_ENV=integration_test ./manage.py run_integration_testing_suite

# Terminal 4: Process testing queue (when prior command completes)
DJANGO_ENV=integration_test ./manage.py process_testing_queue

​
V2 Test

just intg load-customers
This will load the customers and shadow templates specified in maybern/integration_testing_ci_config.json. The feature flags specified in the config will be enabled for the test run.

​
API Integration Tests Deep Dive

​
Adding an API Test

API integration tests are defined in server/apps/integration_testing/private/services/api_testing/api_tests in files roughly named as {app_name}_tests.py (e.g., fees_tests.py). All API tests inherit from GenericAPITest, which is defined in server/apps/integration_testing/private/services/api_testing/api_test_definition.py.
Not all APIs need an API test. We primarily use them to test APIs that operate on top of or read from transaction data. A simple CRUD API that’s a snapshot of a customer configuration probably doesn’t need one.
If you’re intentionally not testing an API, add it to KNOWN_UNTESTED_INTENTIONALLY_SKIPPED_GET_APIS in test_check_all_get_apis_tested.py.

​
Creating an API Test

To add an API test (that does not return GenericTableData):
  1. Create a new class inheriting from GenericAPITest
  2. Specify three generic parameters: the API class itself, the param dataclass, and the response dataclass
  3. Implement the abstract method get_api_test_request_list
All param dataclasses should have a dataclass_name field matching the name of the dataclass and should be added to the GenericAPITestParamData annotated union in the api_test_definition.py file.

​
GenericAPITableTest

Many APIs return GenericTableData, so we introduced GenericAPITableTest which automatically sorts the rows in the table for deterministic ordering. Use this for APIs that return GenericTableData.

​
Post-Processing Responses

Many APIs require post-processing to ensure deterministic results (sorting, rounding, etc.). Override get_api_test_response_content_for_comparison_based_on_status_code and use:
  • _deserialize_response_content to deserialize the API response
  • Process the response
  • serialize_response to reserialize

​
Running API Integration Tests

  1. Store expected results in backend/api-testing as <customer_name>_api_expected_results.json
  2. Copy files from S3 at maybern-dev-infra-config/api-testing, or generate locally:
./manage.py write_api_test_responses_to_json
  1. Run the comparison:
./manage.py compare_api_test_responses_with_expected_json

​
Viewing Failed Test Results

If tests fail, the diffs are output to:
  • backend/api-testing/<customer_name>_api_diff_results.json
  • backend/api-testing/<customer_name>_api_diff_results.yaml
For each failed API request, we output:
  • expected_actual_response_diff
  • actual_expected_response_diff
  • actual_response
  • expected_response
When API tests fail on a branch, diffs are automatically stored in S3 at maybern-dev-infra-config/api-testing/failed-test-results. Use your PR number to locate the diffs for your branch.