Use BYOC Traffic with proxymock
In a bring-your-own-cloud (BYOC) deployment, a collector or the Speedscale Forwarder can write captured traffic to an object-store bucket you own. proxymock pulls historical traffic directly from that bucket into a local workspace, so the import operation does not route the downloaded traffic through Speedscale.
Enabling a BYOC exporter does not automatically disable the Speedscale Cloud exporter. Configure exporters separately when captured RRPairs must remain only in customer-controlled destinations. See Network and data boundaries.
The pull runs locally against your bucket and writes ordinary RRPair files for inspection, mocking, and replay. S3 uses the AWS credential chain; GCS uses Google Application Default Credentials; Azure Blob uses Azure identity or an environment connection string.
There are three ways to run the pull, over the same bucket layout:
- CLI with
proxymock import s3,proxymock import gcs, orproxymock import azureblob(v2.5.1169+). - MCP with the
pull_byoc_buckettool, so an AI agent can fetch its own production context. - Web through the source picker in
proxymock web.
How the bucket is laid out
The BYOC OpenTelemetry awss3 exporter writes OTLP-JSON objects under the byoc/ prefix, in hive-style year=/month=/day=/hour=/minute= partitions. When the bucket contains _speedscale/byoc-layout.json, proxymock reads it to enumerate workload-specific namespace=/app=/… prefixes directly, which makes a narrow pull cheap. The legacy Fluent Bit layout, with objects at the bucket root and hour-granularity partitions, is also supported.
Point --prefix at byoc/ for the current S3 and native GCS collector layouts. The Azure chart uses a different layout. proxymock prunes the object listing by the time window before it downloads anything, so a tight --from/--to keeps the pull fast even against a large bucket.
Direct S3 replay storage layout
The direct customer-owned replay storage mode writes each captured RRPair as JSON under <PREFIX>/records/rrpairs/year=/month=/day=/hour=/minute= partitions. Point --prefix at that path instead of byoc/:
proxymock import s3 --bucket customer-speedscale --prefix speedscale/records/rrpairs/ \
--service checkout --from now-15m --out ./proxymock/byoc-check
For a private S3-compatible service, add --s3-endpoint-url <URL> and --s3-force-path-style if the service requires path-style addressing. The importer uses the time partitions to narrow its object listing. This mode currently writes one object per captured RRPair, so size S3 write throughput and retention for the expected capture volume before using it for a large workload.
Before you begin
proxymockinstalled.- Read access to the BYOC bucket. For S3, use the standard AWS credential chain:
AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY,AWS_PROFILE, or an instance role. For GCS, use Google Application Default Credentials. For Azure Blob, use an Azure identity with Storage Blob Data Reader on the selected container or an authorized environment connection string. Provider setup is described below. No Speedscale account or API key is required for the pull.
Pull with the CLI
Pull matching traffic from the last hour, narrowing by service and status:
proxymock import s3 --bucket my-bucket --prefix byoc/ \
--filter '(service IS "checkout") AND (status IS "500")'
The --filter string uses the Speedscale traffic filter language. Convenience flags (--from, --to, --service, --endpoint, --status, --namespace, --direction, --trace-id) are added only when the filter does not already cover that criterion; where they overlap, the filter wins. --status is an exact match and --endpoint is a substring match, so use --filter for anything more specific.
The imported RRPairs land in proxymock/imported-s3-<timestamp>/ by default. From there, replay or mock them like any recording:
proxymock replay --in ./proxymock/imported-s3-<timestamp> --test-against http://localhost:8080
Test the layout with a local directory
--local-dir reads from a local directory tree with the same layout as a BYOC bucket, which is useful for air-gapped work and for testing a pull without touching S3. When it is set, --bucket and AWS credentials are not used:
proxymock import s3 --local-dir ./fixtures/byoc-traffic --prefix byoc/ --service checkout --limit 10
Redact on the way in
Pass --dlp-config to apply a local DLP config to matched RRPairs before they are written to disk, so sensitive values never land in your workspace unredacted. This is the air-gap-friendly path: the redaction happens locally, against a rule you author locally.
proxymock import s3 --bucket my-bucket --prefix byoc/ --from now-15m --dlp-config my-dlp.json
See Author DLP and filter rules locally for how to build and test that config.
Keep pulling as traffic arrives
--follow keeps the import running, re-listing the bucket for new objects until you interrupt it. Tune the cadence with --poll-interval (default 30s):
proxymock import s3 --bucket my-bucket --prefix byoc/ --service checkout --follow
GCS
Use proxymock import gcs to read directly through the native Google Cloud Storage API. Authenticate with Google Application Default Credentials (ADC): run gcloud auth application-default login for local development, set GOOGLE_APPLICATION_CREDENTIALS to a credentials file, or use workload identity in your runtime. The identity needs storage.objects.list and storage.objects.get on the bucket.
gcloud auth application-default login
proxymock import gcs --bucket my-gcs-bucket --prefix byoc/ \
--service checkout --from now-1h
The native command uses Google credentials and does not need AWS HMAC keys, --region, --s3-endpoint-url, or --s3-force-path-style. It shares the S3 import's layout handling, filters, DLP, and follow mode. Imported RRPairs land in proxymock/imported-gcs-<timestamp>/ by default.
--bucket is the bucket name only. --prefix is an object-key prefix inside that bucket, such as byoc/; do not pass a gs:// URL or include the bucket name. For the legacy Fluent Bit layout with objects at the bucket root, omit --prefix.
S3 interoperability compatibility
For an existing GCS workflow that uses the S3-compatible XML interoperability API, proxymock import s3 remains available. Export the collector chart's GCS HMAC access ID and secret as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. These values are GCS HMAC credentials, not an AWS key pair or a Google service account JSON key.
export AWS_ACCESS_KEY_ID="<GCS_HMAC_ACCESS_ID>"
export AWS_SECRET_ACCESS_KEY="<GCS_HMAC_SECRET>"
unset AWS_SESSION_TOKEN
proxymock import s3 --bucket my-gcs-bucket --prefix byoc/ \
--region auto \
--s3-endpoint-url https://storage.googleapis.com \
--s3-force-path-style \
--service checkout --from now-1h
Keep the endpoint exactly https://storage.googleapis.com; put the bucket name only in --bucket. A bucket-qualified endpoint such as https://my-gcs-bucket.storage.googleapis.com can produce incorrect addressing or TLS errors. Use the native import gcs command when S3 interoperability authentication or requests fail.
Azure Blob Storage
Use proxymock import azureblob in proxymock v2.5.1169 or later to read through the native Azure Blob API. Authenticate on the approved proxymock host with az login, an Azure workload identity, or environment credentials supported by DefaultAzureCredential. The reader needs Storage Blob Data Reader on the selected container. Keep reader credentials separate from the collector's writer credentials.
az login
proxymock import azureblob \
--azure-account-url https://mystorageacct.blob.core.windows.net \
--bucket byoc --service checkout --from now-1h --to now \
--limit 100 --out proxymock/azure-traffic --output json
The account URL contains the HTTPS storage account endpoint only, without a container path or SAS token. --bucket names the container. The reference Azure chart writes blobs under YYYY/MM/DD/logs_* at the container root, so leave --prefix unset unless you configured a matching prefix. The S3 chart's byoc/ prefix does not apply.
For connection-string authentication, set AZURE_STORAGE_CONNECTION_STRING securely in the process environment and omit --azure-account-url. Identity authentication uses --azure-account-url or AZURE_STORAGE_ACCOUNT_URL. Do not put a connection string or SAS token in command arguments, source-controlled values, or chat. Azure account-URL identity and connection-string settings are mutually exclusive.
Azure imports share the traffic filters, DLP, limits, output formats, retries, cancellation, and --follow behavior of the other native importers. A limited viewing sample can omit dependencies or part of a journey; check completeness before replay. Azure cluster-reader discovery is unavailable, so the CLI, web server, or MCP process uses credentials on its own host. These traffic imports are separate from the snapshot-repository --bucket-provider workflows.
Older proxymock clients can upgrade or use the chart repository's azure-gather.py fallback.
S3-compatible stores
For MinIO, DigitalOcean Spaces, or another S3-compatible store, set --s3-endpoint-url and usually --s3-force-path-style. Set --region when the SDK cannot infer it. The full flag list is in the CLI reference.
Pull with the MCP tool
The pull_byoc_bucket MCP tool gives an AI coding assistant the same pull. It runs locally with no Speedscale account; credentials come from the AWS credential chain, Google ADC, or local Azure credentials for the selected provider. Describe what you need and the assistant fills in the parameters:
Pull the last 15 minutes of checkout 500s from the
my-bucketBYOC bucket and replay them against my local build.
Set storage-provider to gcs for Google Cloud Storage or azureblob for Azure Blob; the default is s3. Azure identity uses azure-account-url; connection-string authentication reads AZURE_STORAGE_CONNECTION_STRING from the MCP server environment. The tool takes bucket plus the same narrowing parameters as the CLI (prefix, from, to, service, namespace, status, trace-id, or a full filter), returns the import summary, and writes RRPair files under ./proxymock/ in an imported-s3-<timestamp>, imported-gcs-<timestamp>, or imported-azureblob-<timestamp> directory for the selected provider. This is distinct from pull_remote_recording, which pulls from Speedscale-managed cloud; use pull_byoc_bucket when the traffic lives in your own bucket. See the MCP Tools reference for the full parameter list.
Pull from proxymock web
In proxymock web, the import source picker offers a BYOC bucket as a source alongside local files. Select Google Cloud Storage for a native GCS pull using the proxymock process's Google ADC credentials, or Azure Blob Storage for a native Azure pull using its local Azure credentials. For Azure identity, supply the account URL; leave that field empty when using an environment connection string. Then give it the bucket, prefix, and a time window, and the imported run appears in the Run selector like any other recording, ready to explore in the Requests grid or replay.
Next steps
- Author DLP and filter rules locally redacts or trims the pulled traffic before you use it.
- Explore and Replay Sessions Locally walks the record-and-replay loop the imported traffic feeds into.
- Fix Replay Failures with Recommendations correlates rotating values so imported traffic replays cleanly.
Load-test with Locust
Export to Locust turns downloaded inbound requests into a Locust test. Keep the outbound records to run the application with Speedscale dependency mocks. The guide links to a runnable Kubernetes example.