I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
Ok, done: https://github.com/mseri/zunzuncito
My main focus is systems with very low ram. On my M1 with 8Gb or RAM llama.cpp cannot run gemma4 26b, but this implementation works fine with 5-6 tok/s.
I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have to admit that it works really nicely for me
As much as I agree with the need for digital independence and the fact that universities (and governments) in Europe are over reliant on US tech, it is not as simple as you describe.
There is a lot more happening in the administrative and infrastructural side of things in most universities that one barely observes as student. So every change needs to take also that into account, the management and maintenance of services and infrastructures that must reliably support thousands of users, with relatively strict privacy and security standards, and their migration.
I use this before submission and recommend others do too. If ai was in charge of arXiv Id have it integrated as an optional part of the submission process.
I do this by hand. In the whole process, cleaning up the TeX before submission is a small step. And I like to keep some comments, like explaining how some Tikz figures are made. Might help someone some day.
META did pirate basically all books in Anna’s archive but if I remember correctly they just whispered a a cried sorry and it ended up as that. Why are they also not asked to pay?
True, but we should also remember that some services like the fast responses and the image generations (may?) run in US data centres also for Mistral. So that part of the data, in principle, may end up in the ends of other extra European countries.
This said, I am really supportive of Mistral, like their work, and hope that they will get more recognition and more EU-centric institutional support.
> By submitting User Submissions through the Services, you hereby do and shall grant Inception a worldwide, non-exclusive, perpetual, royalty-free, fully paid, sublicensable and transferable license to use, edit, modify, truncate, aggregate, reproduce, distribute, prepare derivative works of, display, perform, and otherwise fully exploit the User Submissions in connection with this site, the Services and our (and our successors’ and assigns’) businesses, including without limitation for promoting and redistributing part or all of this site or the Services (and derivative works thereof) in any media formats and through any media channels (including, without limitation, third party websites and feeds), and including after your termination of your account or the Services. For clarity, Inception may use User Submissions to train artificial intelligence models. (However, we will not train models using submissions from users accessing our Services via OpenRouter.)
I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available