Prove the deploy by digest, not by tag
The preceding commit stops `set image` being a silent no-op. This asserts the outcome: after the rollout, the tag is resolved to a digest at the registry and th...
A rebuild of the same commit deployed nothing and reported success
The image is tagged with the commit SHA, so rebuilding the same commit produces an identical image reference. `kubectl set image` then changes nothing, the Depl...
Let a rollout finish what the old pod was doing
Deploys replaced the only pod of each service with nothing to catch the requests in flight. Three settings, applied across the fleet: - preStop sleeps 10s befo...
A KB link goes to the article its text names
A member asked how to set their org up and got the setup checklist back, each step naming the article that explains it. All three links pointed at the same arti...
No upstream error text reaches a member, whichever provider failed
memberTextFor fell through to getMessage() for anything that was not an AiProviderException. Only the HuggingFace adapter classifies its refusals; every other o...
Retire a model the provider won't route, and stop quoting it at members
Follow-up to the router-catalogue fix. That stops unroutable models being LISTED; this stops one already in an org's catalogue from breaking the assistant and n...
List the Hugging Face models the router will actually serve
Chat goes to the Inference Providers router **************** but discoverModels asked the HUB for models that are "warm" somewhere — a different and much larger...
Remove the unreviewed-reply unread-count endpoints
Paired with the kamo-internal removal of the AI unread badge: the count never told the member anything actionable (no push channel backs it, so non-zero only ev...
Size the pod above its own JVM heap ceiling [skip ci]
The image starts the JVM with -XX:MaxRAMPercentage=70 -XX:+AlwaysPreTouch, so the heap alone may take 70% of the container limit and pre-touch keeps every commi...
Administering models and reading usage take the rights that name them
AiModelController and AiUsageController checked organization ownership and no right. The 403s already in the model controller compare provider.getOrganizationId...
Declare the object-storage secret the ConfigMap no longer carries
The MinIO key was moved out of the ConfigMap into the minio-app-credentials secret, but the envFrom that supplies it was only ever applied to the live deploymen...
Stop shipping a shared credential as a config default
The platform's shared password was baked in as the fallback for DB_PASSWORD and MINIO_ROOT_PASSWORD, so it lived in this repo and in the built image as well as ...
Scan the shared MFA repositories
MfaEnrollmentService is a @Service in kamo-shared-library, so the wildcard @ComponentScan("com.kamo") in this application constructs it whether or not this serv...
Harden the kubectl download against flaky egress [skip ci]
dl.k8s.io over the runner's egress intermittently drops mid-transfer: curl: (56) OpenSSL SSL_read: decryption failed or bad record mac which fails the deploy ...
Classify vision failures so the retry loop and circuit breaker actually work
Two bugs in visionComplete, both of which defeated machinery built to handle them. A blank-content response was returned as SUCCESS and ended the retry loop. E...
Never route vision to an adapter that cannot send an image
AiProvider.supportsVision is a hand-editable database flag, and routing trusted it alone. The HuggingFace and Cohere adapters never read imageUrls at all, and t...
Anthropic dropped every system message but the last
buildRequestBody assigned rather than accumulated the system field, so with more than one system message only the final one reached the model. In practice that ...
Locked rules survive a section or whole-document override
An author taking over a rule section by hand replaced the entire section body, which silently discarded the locked pack rules inside it - so 'never quote a rate...
Streamed replies were dropped, persisting blank assistant messages
Every successful chat completion came back as an EMPTY assistant bubble. The assistant rows were being saved with content_len=0 and is_error=false, so there was...
Try the next model when a vision model rejects the image
Probing production again after the modality fix, routing moved from a text-to-speech model to o3-mini — an o-series reasoning model that also takes no image inp...
Stop routing vision to text-to-speech models
Found by probing production: org 1188756624575266822's vision calls resolved to gpt-4o-mini-tts-2025-03-20 — a TEXT-TO-SPEECH model — so every vision call retur...
Load history images via query, not lazy getImages() (WS path)
**************** runs on the WebSocket path (called from AiChatWebSocketHandler), which is NOT an HTTP request, so open-session-in-view does not apply. Iteratin...
Allow SWITCH_MODEL to reset a session to Auto
handleSwitchModel always UUID-parsed modelId and errored on null, so a session could never be reverted to automatic model selection mid-chat. Parse modelId as a...
Robust vision provider/model selection across vendors
Two-pass selectVisionProvider: prefer any org provider that has an active vision-capable model adapter already emits image_url content and reports supportsVisio...
Use cluster MinIO service DNS; drop deprecated WireGuard/CGNAT IPs and retired k2m1/k0m1
Lower RAG min-score threshold from 0.5 to 0.25 to stop filtering relevant KB results
The 0.5 threshold was silently discarding valid results from the embedding model, especially for domain-specific content with smaller 384-dim vectors.
Add response body and request logging to RagClient for debugging
Log the RAG search request URL, orgId, query, and response body on error to diagnose 500 responses from RAGService.
Use non-streaming chat completion for REST endpoint
Streaming over REST with blockLast() was producing empty responses because WebClient's SSE parsing doesn't work reliably with bodyToFlux(String.class). Added ha...
Add detailed error logging to HuggingFaceAdapter for debugging 400 errors
Captures and logs the actual HF error response body instead of just status code.
Switch HuggingFace to router.huggingface.co (api-inference is deprecated)
HF returns 410 on api-inference.huggingface.co with message to use router.huggingface.co instead.
Use HuggingFace OpenAI-compatible endpoint with model in request body
HF's /v1/chat/completions endpoint takes the model ID in the JSON body, not in the URL path. Switched from per-model URL routing to standard
Add model count to provider list, switch HF to router endpoint
- Include modelCount in provider list GET response - Switch Hugging Face default base URL to router.huggingface.co - Support both api-inference and router URL f...
Return full decrypted API key in provider responses
Users need to see and edit the full key, not a masked version.
Normalize base URL to prevent /v1/v1 duplication
Strips trailing /v1 from base URL before appending /v1/chat/completions, so users can enter base URLs with or without the /v1 suffix.
Restructure test connection to discover models first, handle rate limits gracefully
- Model discovery (lightweight GET) runs before chat test (POST) - Rate limit (429) errors show user-friendly warning instead of failing - Auth errors (401/403)...
Use provider-specific test model instead of hardcoded gpt-4o-mini
Each provider type now uses an appropriate model for connection testing
Serialize entity IDs as strings to prevent JavaScript precision loss
CockroachDB unique_rowid() generates INT8 values that exceed JavaScript's Number.MAX_SAFE_INTEGER, causing ID corruption in the frontend.
Add OTK authentication support matching SecurityService pattern
- Add OTKPreAuthFilter to validate X-OTK header and resolve to *** session - Update SessionHelper to read from request attributes set by filter - Update all con...
Like what you see shipping?
Every one of these updates lands in your workspace automatically. Start free and watch it grow week after week.