upload-release-action | Upload files to a GitHub release | File Upload library

As you've implied, you can't limit the scope of a personal access token to different scopes for different repos. Theres a few ways of doing this.

Intermediate, public storage

The first is to upload the artifacts to an intermediate place, accessible from anywhere, e.g. Dropbox, Docker Hub, etc. Then you can manually trigger a github action in your public repo to pull this artifact back down and create a release from it. To manually trigger this action you could use the repository_dispatch event either using cURL / postman locally (with an access token auth bearer) or using something like https://www.actionspanel.app/ which is a github app which allows you to manually trigger github actions using repository_dispatch, with parameters so your download link would be a parameter.

Personal access token

The simplest option is still a personal access token though. Your workflow above has repo_token: XXXXXXXXX which makes me wonder if you know about github secrets? Ideally this token would be stored in a secret then accessed using ${{ secrets.BRANDONS_TOKEN }}. I would ask why you are worried about a personal access token. If you use github secrets and are careful about the 3rd party code you pass the token to (you may not want to simply pass your token to @master, for example), it should be fine.

GitHub Apps & Webhooks

GitHub apps or webhooks would be another way, you can authenticate those on a per-person basis and per-repo basis but you'd need an application running online to receive and parse the messages and its quite a big piece of work.

(Probably not) GitHub Deploy Keys

Another thing to be aware of is Github Deploy Keys, you can use these to obtain read/write access to a single repository without an account attached. You would then store this deploy key in a secret in the settings of the other repo. However, I'm not sure you can trigger releases with deploy keys - they are not bound to an account so I'm unsure who's username would be visible on the release history.