Spaces:
Running
Running
Commit
Β·
a205fc5
1
Parent(s):
e18ea9a
fix: Resolve Free Tier synthesis issue with server-side API keys
Browse files- Marked the bug regarding Free Tier synthesis using server-side API keys as resolved.
- Implemented a new `synthesize()` method in `HFInferenceJudgeHandler` to ensure proper usage of the free HuggingFace Inference API.
- Updated orchestration logic to check for the presence of the `synthesize` method, allowing for correct fallback behavior.
- Enhanced unit and integration tests to verify the new functionality and ensure consistent behavior across different tiers.
- Documented the resolution and verification steps in the relevant markdown files.
docs/bugs/ACTIVE_BUGS.md
CHANGED
|
@@ -13,31 +13,6 @@
|
|
| 13 |
|
| 14 |
## P1 - Important
|
| 15 |
|
| 16 |
-
### P1 - Narrative Synthesis Falls Back to Template
|
| 17 |
-
**File:** `archive/P1_NARRATIVE_SYNTHESIS_FALLBACK.md`
|
| 18 |
-
**Related:** SPEC_12 (implemented but falling back)
|
| 19 |
-
|
| 20 |
-
**Problem:** Users see bullet-point template output instead of LLM-generated narrative prose.
|
| 21 |
-
**Root Cause:** Any exception in LLM synthesis triggers silent fallback to template.
|
| 22 |
-
**Impact:** Core value proposition (synthesized reports) not delivered.
|
| 23 |
-
**Fix Options:**
|
| 24 |
-
1. Surface errors to user instead of silent fallback
|
| 25 |
-
2. Configure HuggingFace Spaces secrets with API keys
|
| 26 |
-
3. Add synthesis status indicator in UI
|
| 27 |
-
|
| 28 |
-
---
|
| 29 |
-
|
| 30 |
-
## P3 - Architecture/Enhancement
|
| 31 |
-
|
| 32 |
-
### ~~P3 - Missing Structured Cognitive Memory~~ FIXED (Phase 1)
|
| 33 |
-
**File:** `P3_ARCHITECTURAL_GAP_STRUCTURED_MEMORY.md`
|
| 34 |
-
**Spec:** [SPEC_07_LANGGRAPH_MEMORY_ARCH.md](../specs/SPEC_07_LANGGRAPH_MEMORY_ARCH.md)
|
| 35 |
-
**PR:** [#72](https://github.com/The-Obstacle-Is-The-Way/DeepBoner/pull/72)
|
| 36 |
-
|
| 37 |
-
**Problem:** AdvancedOrchestrator uses chat-based state (context drift on long runs).
|
| 38 |
-
**Solution:** Implemented LangGraph StateGraph with explicit hypothesis/conflict tracking (`src/agents/graph`).
|
| 39 |
-
**Status:** β
Memory layer built. β³ Integration pending (SPEC_08).
|
| 40 |
-
|
| 41 |
### P1 - Memory Layer Not Integrated (Post-Hackathon)
|
| 42 |
**Issue:** [#73](https://github.com/The-Obstacle-Is-The-Way/DeepBoner/issues/73)
|
| 43 |
**Spec:** [SPEC_08_INTEGRATE_MEMORY_LAYER.md](../specs/SPEC_08_INTEGRATE_MEMORY_LAYER.md)
|
|
@@ -46,17 +21,19 @@
|
|
| 46 |
**Solution:** Extract memory into shared service, integrate into Simple and Advanced modes.
|
| 47 |
**Status:** Spec written. Blocked until post-hackathon.
|
| 48 |
|
| 49 |
-
### P3 - Ephemeral Memory (No Persistence)
|
| 50 |
-
**File:** `P3_ARCHITECTURAL_GAP_EPHEMERAL_MEMORY.md`
|
| 51 |
-
|
| 52 |
-
**Problem:** ChromaDB uses in-memory client despite `settings.chroma_db_path` existing.
|
| 53 |
-
**Solution:** Switch to `PersistentClient(path=settings.chroma_db_path)`.
|
| 54 |
-
**Status:** Quick fix identified, not yet implemented.
|
| 55 |
-
|
| 56 |
---
|
| 57 |
|
| 58 |
## Resolved Bugs
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
### ~~P0 - Synthesis Fails with OpenAIError in Free Mode~~ FIXED
|
| 61 |
**File:** `docs/bugs/P0_SYNTHESIS_PROVIDER_MISMATCH.md`
|
| 62 |
**Found:** 2025-11-30 (Code Audit)
|
|
|
|
| 13 |
|
| 14 |
## P1 - Important
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
### P1 - Memory Layer Not Integrated (Post-Hackathon)
|
| 17 |
**Issue:** [#73](https://github.com/The-Obstacle-Is-The-Way/DeepBoner/issues/73)
|
| 18 |
**Spec:** [SPEC_08_INTEGRATE_MEMORY_LAYER.md](../specs/SPEC_08_INTEGRATE_MEMORY_LAYER.md)
|
|
|
|
| 21 |
**Solution:** Extract memory into shared service, integrate into Simple and Advanced modes.
|
| 22 |
**Status:** Spec written. Blocked until post-hackathon.
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
---
|
| 25 |
|
| 26 |
## Resolved Bugs
|
| 27 |
|
| 28 |
+
### ~~P0 - Free Tier Synthesis Incorrectly Uses Server-Side API Keys~~ FIXED
|
| 29 |
+
**File:** `docs/bugs/P1_SYNTHESIS_BROKEN_KEY_FALLBACK.md`
|
| 30 |
+
**Found:** 2025-11-30 (Testing)
|
| 31 |
+
**Resolved:** 2025-11-30
|
| 32 |
+
|
| 33 |
+
- Problem: Simple Mode crashed with "OpenAIError" on HuggingFace Spaces when user provided no key but admin key was invalid.
|
| 34 |
+
- Root Cause: Synthesis logic bypassed the Free Tier judge and incorrectly used server-side keys via `get_model()`.
|
| 35 |
+
- Fix: Implemented `synthesize()` in `HFInferenceJudgeHandler` to use free HuggingFace Inference, ensuring consistency with the judge phase.
|
| 36 |
+
|
| 37 |
### ~~P0 - Synthesis Fails with OpenAIError in Free Mode~~ FIXED
|
| 38 |
**File:** `docs/bugs/P0_SYNTHESIS_PROVIDER_MISMATCH.md`
|
| 39 |
**Found:** 2025-11-30 (Code Audit)
|
docs/bugs/P1_SYNTHESIS_BROKEN_KEY_FALLBACK.md
CHANGED
|
@@ -1,11 +1,42 @@
|
|
| 1 |
# P0 - Free Tier Synthesis Incorrectly Uses Server-Side API Keys
|
| 2 |
|
| 3 |
-
**Status:**
|
| 4 |
**Priority:** P0 (Breaks Free Tier Promise)
|
| 5 |
**Found:** 2025-11-30
|
|
|
|
| 6 |
**Component:** `src/orchestrators/simple.py`, `src/agent_factory/judges.py`
|
| 7 |
|
| 8 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
|
| 10 |
When using Simple Mode (Free Tier) without providing a user API key, users see:
|
| 11 |
|
|
|
|
| 1 |
# P0 - Free Tier Synthesis Incorrectly Uses Server-Side API Keys
|
| 2 |
|
| 3 |
+
**Status:** RESOLVED
|
| 4 |
**Priority:** P0 (Breaks Free Tier Promise)
|
| 5 |
**Found:** 2025-11-30
|
| 6 |
+
**Resolved:** 2025-11-30
|
| 7 |
**Component:** `src/orchestrators/simple.py`, `src/agent_factory/judges.py`
|
| 8 |
|
| 9 |
+
## Resolution Summary
|
| 10 |
+
|
| 11 |
+
The architectural bug where Simple Mode synthesis incorrectly used server-side API keys has been fixed.
|
| 12 |
+
We implemented a dedicated `synthesize()` method in `HFInferenceJudgeHandler` that uses the free
|
| 13 |
+
HuggingFace Inference API, consistent with the judging phase.
|
| 14 |
+
|
| 15 |
+
### Fix Details
|
| 16 |
+
|
| 17 |
+
1. **New Feature**: Added `synthesize()` method to `HFInferenceJudgeHandler` (and `JudgeHandler` protocol).
|
| 18 |
+
- Uses `huggingface_hub.InferenceClient.chat_completion` (Free Tier).
|
| 19 |
+
- Mirrors the `assess()` logic for consistent free access.
|
| 20 |
+
|
| 21 |
+
2. **Orchestrator Logic Update**:
|
| 22 |
+
- `SimpleOrchestrator` now checks `if hasattr(self.judge, "synthesize")`.
|
| 23 |
+
- If true (Free Tier), it calls `judge.synthesize()` directly, skipping `get_model()`/`pydantic_ai`.
|
| 24 |
+
- If false (Paid Tier), it falls back to the existing `pydantic_ai` agent flow using `get_model()`.
|
| 25 |
+
|
| 26 |
+
3. **Test Coverage**:
|
| 27 |
+
- Updated `tests/unit/orchestrators/test_simple_synthesis.py` to mock `judge.synthesize`.
|
| 28 |
+
- Added new test case ensuring Free Tier path is taken when available.
|
| 29 |
+
- Fixed integration tests to simulate Free Tier correctly.
|
| 30 |
+
|
| 31 |
+
### Verification
|
| 32 |
+
|
| 33 |
+
- **Unit Tests**: `tests/unit/orchestrators/test_simple_synthesis.py` passed (7/7).
|
| 34 |
+
- **Integration**: `tests/integration/test_simple_mode_synthesis.py` passed.
|
| 35 |
+
- **Full Suite**: `make check` passed (310/310 tests).
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## Symptom (Archive)
|
| 40 |
|
| 41 |
When using Simple Mode (Free Tier) without providing a user API key, users see:
|
| 42 |
|