Live scratch-org tests

Offline documentation and contract checks

The normal, credential-free test suite includes tests/test_documentation_examples.py and tests/test_documented_contracts.py. It compiles Python blocks in the README and documentation plus the scripts under examples/, checks toolkit imports, and executes selected complete examples and fragments against in-memory HTTPX transports. CLI authentication is replaced with synthetic tokens, and live sockets/process execution are blocked in those example tests. The same checks run in public CI as part of the ordinary test suite.

.venv/bin/python -m pytest tests/ -m 'not integration' --no-cov -q

Executed examples cover sync/async clients, model hydration/defaults, query construction and result copies, and multipage Bulk Query streaming. Separate contract tests cover pagination failures and caller-owned client reuse. Syntax and mocked execution do not establish Salesforce acceptance, permissions, or org-specific schemas for every snippet. Live checks below remain explicitly opt-in and require authorization before provisioning resources.

Scratch-org acceptance suite

The scratch-org acceptance suite uses your authenticated Salesforce CLI Dev Hub to create a new one-day Developer scratch org, deploy disposable test components, and delete the org at session teardown. It never uses the default target org for API tests or deployments. It does consume one active scratch-org slot and one creation from your Dev Hub’s daily allocation.

Prerequisites

  • Install project development dependencies and the current sf CLI.

  • Authenticate a Dev Hub with scratch-org creation permissions and available quota. Existing CLI authentication is reused; no secrets belong in test files.

  • Run serially, without pytest-xdist. The fixture refuses parallel workers.

Run only the scratch suite:

.venv/bin/python -m pytest tests/integration/scratch \
  --with-integration --scratch-org --dev-hub my-dev-hub --no-cov -v

--dev-hub is optional if target-dev-hub is configured or exactly one connected authenticated Dev Hub is discoverable. Ambiguous selection fails rather than choosing an ordinary/default org. Both --with-integration and --scratch-org are required. Normal local/CI test runs do not provision orgs. No CLI default org/Dev Hub configuration is changed.

Coverage and safety

The same three acceptance checks run with synchronous and asynchronous clients:

  • Custom Metadata read/copy/update/re-read, preserving a typed boolean field and restoring the original label. This exercises the real SOAP namespace layout and QName-valued metadata fields through the repaired serializer.

  • Missing Custom Metadata record behavior.

  • Tooling API fetch/delete of a scratch-only Apex class deployed by the fixture.

The temporary seed project and clients use API 66.0. The org must advertise that API version; scratch org creation itself uses the CLI’s supported default.

Before seeding, the CLI identity must match the org ID returned by scratch-org creation. Before API tests, the toolkit’s authenticated organization ID must match it too. Every deployment, login and deletion specifies the newly created username explicitly; no API writes target the Dev Hub. Seed metadata and CLI project state stay in pytest’s temporary directory, not this repository.

The fixture attempts deletion even when seeding or tests fail, but only after verifying the created org’s identity during setup and again immediately before deletion. An ambiguous creation failure, identity mismatch, or failed identity check suppresses automatic deletion and reports manual-cleanup guidance; a generated alias alone is not proof of ownership. Deletion failures are reported, not silently ignored.

If both the seed deployment and cleanup fail, diagnostics retain the earlier sanitized setup error as an exception note. Raw test exceptions and CLI cleanup output are not added to teardown diagnostics.

If automatic cleanup is refused, pytest is force-killed, the machine disconnects, or creation is still pending, inspect sf org list and independently verify the created test org’s identity before deleting only that org with sf org delete scratch --target-org <verified-target> --no-prompt. The reported sf-toolkit-test-... alias helps locate it, but must not replace the identity check. A pending creation may need sf org resume scratch first. One-day expiration is a backstop, not confirmation of deletion.

CLI output and credentials are captured in memory, not printed in test errors. Scratch-client login uses the same sanitized CLI adapter for org display and, when necessary, access-token retrieval; raw CLI failure messages are omitted. Do not enable request/response body logging, CLI tracing or pytest --showlocals when running authenticated tests. Temporary CLI projects may contain Salesforce-managed auth references; handle them as local private data.

These tests establish behavior for the disposable Custom Metadata and Apex components in this scratch-org configuration. They do not validate External Client App certificate renewal, JWT audience/app policies, production security settings, every metadata type, or response compression/retry failure injection. Those still require their own targeted acceptance tests.