Bedrock endpoint, modelSpecs, Agents Builder, cross-region inference, and so on

[toarni]

First of all thanks for this wonderful project and all the hard work already done till this point. I really love LibreChat and I'm rolling it out here at my company.

And also sorry if this looks like a dumb question, but I searched all issues discussions, found several solutions but some seem outdated and I can not achieve what I want/need.

Prereq

• Using LibreChat 0.8.7-rc1 on an AWS EC2 machine in eu-west-3
• Using bedrock as only AI endpoint for now
• Most of the models are eu-infered models like eu.anthropic.claude-haiku-4-5-20251001-v1:0
• But also using some region specific models like zai.glm-4.7-flash (eu-central-1) or zai.glm-5 (eu-north-1)
• Default bedrock invoke region is eu-central-1

Chapter 1 - First Steps with LibreChat

Endpoint Config

  bedrock:
    inferenceProfiles:
      'zai.glm-5': "arn:aws:bedrock:eu-north-1:123xxx123:application-inference-profile/1xyz1"
    availableRegions:
      - eu-central-1
      - eu-west-3
      - eu-north-1
    models:
      - "eu.anthropic.claude-haiku-4-5-20251001-v1:0"
      - "zai.glm-5"
      - "zai.glm-4.7-flash"

"Problems" with this setup per model

• Haiku: Chat works fine also contextSize is ~170K which looks ok to me, as Haiku should have ~200K
• GLM 4.7 Flash: Chat works fine, but context window is ~28K, which should be 203K
• GLM 5: Chat works only if I set the region to eu-north-1 in the model settings. Same for the context window like GLM 4.7 Flash
  • So here the inferenceProfile Mapping seems not to work? Am I doing something wrong?
    I checked the whole docs, but I only get an unknown model invalid error here. Everything else (permissions in AWS) is set up correctly, as I made it work via the modelSpecs using the full ARN without the region.

Chapter 2 - Second Config Evolution

After having those problems I tried to solve them via the modelSpecs which kind of worked.

ModelSpec Config

modelSpecs:
  enforce: false
  prioritize: true
  addedEndpoints: ["bedrock"] # -> only here to be able to also invoke the models directly in the ChatUI
  list:
    - name: "plain-claude-haiku-4-5"
      group: "Anthropic"
      label: "Claude Haiku 4.5"
      showIconInMenu: true
      showIconInHeader: true
      description: "200K, 1.1, 5.5 | Fast & cheap, simple tasks"
      preset:
        endpoint: "bedrock"
        model: "eu.anthropic.claude-haiku-4-5-20251001-v1:0"
        modelLabel: "Claude Haiku 4.5"
        maxContextTokens: 200000

    - name: "plain-glm-5"
      group: "GLM"
      label: "GLM 5"
      showIconInMenu: true
      showIconInHeader: true
      description: "203K, 1.20, 3.84 | Next-gen GLM model"
      preset:
        endpoint: "bedrock"
        model: "arn:aws:bedrock:eu-north-1:123xxx123:application-inference-profile/1xyz1"
        modelLabel: "GLM 5"
        maxContextTokens: 203000

    - name: "plain-glm-4-7-flash"
      group: "GLM"
      label: "GLM 4.7 Flash"
      showIconInMenu: true
      showIconInHeader: true
      description: "203K, 0.08, 0.48 | Very fast, very cheap"
      preset:
        endpoint: "bedrock"
        model: "zai.glm-4.7-flash"
        modelLabel: "GLM 4.7 Flash"
        maxContextTokens: 203000

With these modelSpecs in place, I was able to solve the problems with the "wrong" contextSize and with the "not working" inferenceProfile mapping for GLM 5.
When I now removed the addedEndpoints from this config everything looked and worked as expected.

Chapter 3 - Adding Agents

Now I also wanted to add the agentBuilder, which is one of the most powerful features in LibreChat.
So, I added addedEndpoints: ["agents"] to the modelSpecs, which opened the Agents-Selection and Builder. So far so good.

But now I'm facing the problem that I can only select the "raw" models from the bedrock endpoint. So again I have the same problems as in Chapter 1. Wrong contextSize and "wrong" region for GLM 5. I'm aware that I can set the config for each agent I create, but I can not expect from all my colleagues here that they are aware of the correct contextSizes and regions, which they need to set up in their agent to work properly.

I already tried so many "solutions" to somehow solve this, but I couldn't find a way.

Conclusion

It looks to me that:

1. I'm doing something fundamentally wrong in my bedrock endpoint config, which causes the inferenceProfile mapping not to work and which causes the wrong ContextSizes.
2. or that it's not possible to achieve what I want right now

What I want to achieve

1. Setting up some models with custom parameters (correct contextSize, region, pricing, ...), which could be used directly to chat with.
2. Letting users create their own agents but let them only choose from the models defined in 1.

To achieve that, I need the agentBuilder to be able to choose from the "modelSpecs" or to be able to set model-specific configurations directly in the bedrock endpoint. But I can not make any of those two ways work.

I was thinking about the solution to group my models by region into custom endpoints. With the new tokenConfig option there, I should be able to achieve everything. But thinking this into the future I need a custom endpoint for every region I want to use, which seems to be really annoying.

Really appreaciating any help or advice, which points me into the right direction.

[kimnamu]

Thanks for the detailed write-up, @toarni. The two configs plus the per-model observations made both behaviours traceable from the code alone.

1. GLM context shows ~28K on the bedrock endpoint. The Bedrock endpoint resolves context size from bedrockModels only, and that map has no Z.AI/GLM entries. The glm-4.7-flash: 202752 and glm-5: 204800 values live in aggregateModels, which serves the openAI/agents/custom endpoints. With no match the endpoint falls back to system_default: 32000, and after the output budget is reserved that is the ~28K you see. Haiku is fine because anthropicModels is spread into bedrockModels.

Endpoint	Lookup for zai.glm-4.7-flash
bedrock (bedrockModels)	no match, 32K default
agents, or a modelSpec on agents (aggregateModels)	glm-4.7-flash = 202752

Workaround today: set maxContextTokens in the modelSpec preset (it is a preset field), or run the model through an agent. The real fix is adding zai.* entries to bedrockModels; happy to open that PR if a maintainer wants it.

2. zai.glm-5 only works with the region set. inferenceProfiles copies the ARN into applicationInferenceProfile, but the client region stays model region ?? BEDROCK_AWS_DEFAULT_REGION; nothing reads the region out of the ARN. Your eu-north-1 profile is therefore sent to bedrock-runtime.eu-central-1, which cannot resolve it, and that surfaces as the model invalid error. Setting the region per model, as you did, is currently the intended path. Deriving it from the ARN would be a small follow-up.

Code references (main, read 2026-09-14)

packages/api/src/utils/tokens.ts:415-425     const bedrockModels = { ...mistralModels, ...cohereModels, ...deepseekModels, ...moonshotModels, ...metaModels, ...ai21Models, ...amazonModels, ...openAIBedrockModels, ...anthropicModels }
packages/api/src/utils/tokens.ts:451-465     const aggregateModels = { 'glm-4.7-flash': 202752, 'glm-5': 204800, ..., ...bedrockModels, ...openAIModels }
packages/api/src/utils/tokens.ts:492         [EModelEndpoint.bedrock]: bedrockModels
packages/api/src/utils/tokens.ts:520         system_default: 32000
packages/api/src/endpoints/bedrock/initialize.ts:252-254   llmConfig.applicationInferenceProfile = extractEnvVariable(bedrockConfig.inferenceProfiles[model])
packages/api/src/endpoints/bedrock/initialize.ts:266,282   region: (llmConfig.region as string) ?? BEDROCK_AWS_DEFAULT_REGION
packages/data-schemas/src/schema/preset.ts:48              maxContextTokens?: number

The same split exists in v0.8.7-rc1 (GLM only in the aggregate map), so upgrading alone does not change the 28K. I read the code rather than running your exact config, so if the numbers on your instance disagree, say so.