We sincerely apologize for the disruption caused by elevated 5XX error rates impacting the APK registry backend.

Since enabling package deduplication, memory utilization has stabilized between 10–30%, and 5XX error rates have returned to near-zero levels, with only occasional brief spikes observed. Overall service health has significantly improved, and we are continuing to monitor the platform closely to ensure stability.

Thank you for your patience while we worked through this issue.

The overall error rate and impact on CI workflows is now much lower and less disruptive for affected customers.

We are actively monitoring error and memory metrics to confirm sustained stability before declaring a full resolution.

We've deployed the APKINDEX LRU cache update to production, which has led to a significant drop in 503 errors and error-rate spikes for APK downloads from virtualapk.cgr.dev.

Some 503s are still being observed because the cache size appears to be larger than ideal, and the average entry size was underestimated. However, overall service reliability has improved and error rates are much lower than before the change.

We will lower the cache size to better reflect the average cache entry size and lower the rate of 503s even further today.

We are investigating intermittent failures affecting virtualapk.cgr.dev, where some APK index and package fetch requests may return temporary errors during short periods of elevated load.
We have identified the root cause and are rolling out mitigations to improve stability. During impacted windows, retries typically succeed.
The most reliable workaround is having retry-on-failure logic around CI/CD jobs. Reruns almost always succeed once the backend is healthy.

We will provide additional updates as mitigation rollout progresses.