Been working on it on and off for 4 years now. I used to lead an ML consultancy and one thing I've learned is that data scientist DO NOT understand reverse proxies, nvidia drivers, docker and can not set up complex environments for their projects, especially ones that would support collaborative work.
It's been really fun to work on this, currently working on Kubernetes support.
Oh this is awesome, will take a look. I’ve had rough experiences with Kubeflow in the past, recently I’ve been using Coder workspaces https://coder.com/ and it’s pretty good but less ML/AI integrated (but, stitched together with Ray autoscaling nodes it can get really nice)
Yeah, Kubeflow can be rough, we've been down that road too as well.
I really like Coder, but one thing for the life of me I couldn't figure out is that even with the self-hosted version, why does anything have to go through their infra?
With trailer.dev our explicit goal is to have a fulfilling user experience for the following scenario:
- You have servers you own or rent
- You have a team that want some persistent developer workspaces that are running on that server
- You run our 30mb + the server side somewhere and boom, you can have your team(s) work on your project.
If you want we (the trailer team) can run the server component for you, but that's it.
I tried spinning up a coder instance just now again, with docker, and I'm being tunneled through US East Pittsburgh with 117ms of latency for a local dev environment tool :(
(joining very late to the thread, but for future readers)
Coder requires zero internet-access, many of our customers run entirely airgapped. We _currently_ create an HTTP tunnel for first-time users to their Coder instance for a smooth setup. This removes the need to expose their deployment for workspace infra (e.g. EC2 VMs) just to try it out.
We could communicate this more clearly in our UX, which I'll fix up!
In the tweet Jeff Dean says that Cheng at al. failed to follow the steps required to replicate the work of the Google researchers.
Specifically:
> In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), robbing our learning-based method of its ability to learn from other chip designs
But in the Circuit Training Google repo[1] they specifically say:
> Our results training from scratch are comparable or better than the reported results in the paper (on page 22) which used fine-tuning from a pre-trained model.
I may be misunderstanding something here, but which one is it? Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction?
Also, the UCSD group had to reverse-engineer several steps to reproduce the results so it seems like the paper's results weren't reproducible by themselves.
Markov’s paper also has links to Google papers from two different sets of authors that shows minimal advantage of pretraining. And given the small number of benchmarks using a pretrained model from Google whose provenance is not known would be counterproductive. Google likely trained it on all available benchmarks to regurgitate the best solutions of commercial tools.
Training from scratch could presumably mean including the new design attempts and old designs mixed in.
So no contradiction: pretrain on old designs then finetune on new design, vs train on everything mixed together throughout. Finetuning can cause catastrophic forgetting. Both could have better performance than not including old designs.
> Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction?
The Circuit Training repo was just going through an example. It is common for an open-source repo to describe simple examples for testing / validating your setup --- that does not mean this is how you should get optimal results in general. The confusion may stem from their statement that, in this example, they produced results that were comparable with the pre-trained results in the paper. This is clearly not a general repudiation of pre-training.
If Cheng et al. genuinely felt this was ambiguous, they should have reached out to the corresponding authors. If they ran into some part of the repo they felt they had to "reverse-engineer", they should have asked about that, too.
"These major methodological differences unfortunately invalidate Cheng et al.’s comparisons with and conclusions about our method. If Cheng et al. had reached out to the corresponding authors of the Nature paper[8], we would have gladly helped them to correct these issues prior to publication[9].
[8] Prior to publication of Cheng et al., our last correspondence with any of its authors was in August of 2022 when we reached out to share our new contact information.
[9] In contrast, prior to publishing in Nature, we corresponded extensively with Andrew Kahng, senior author of Cheng et al. and of the prior state of the art (RePlAce), to ensure that we were using the appropriate settings for RePlAce."
That is misleading. The first two authors left Google in August 2022 under unclear circumstances. The code and data were owned by Google, that's probably why Kahng continued discussibg code and data with his Google contacts. He received clear answers from several Google employees, so if they were at fault, Google should apologize rather than blame Cheng and Kahng.
"Prior to publication of Cheng et al., our last correspondence with any of its authors was in August of 2022 when we reached out to share our new contact information."
You don't stop being the corresponding authors of a paper when you change companies,
and whatever "unclear circumstances" you imagine took place when they left, they were also re-hired later, which a company would only do if they were in good standing.
In any case, those "Google contacts" also expressed concerns with how Cheng et al. were doing their study, which they ignored:
3.4 Cheng et al.’s Incorrect Claim of Validation by Google Engineers
Cheng et al. claimed that Google engineers confirmed its technical correctness, but this is untrue. Google engineers (who were not corresponding authors of the Nature paper) merely confirmed that they were able to train from scratch (i.e. no pre-training) on a single test case from the quick start guide in our open-source repository. The quick start guide is of course not a description of how to fully replicate the methodology described in our Nature paper, and is only intended as a first step to confirm that the needed software is installed, that the code has compiled, and that it can successfully run on a single simple test case (Ariane).
In fact, these Google engineers share our concerns and provided constructive feedback, which was not addressed. For example, prior to publication of Cheng et al., through written communication and in several meetings, they raised concerns about the study, including the use of drastically less compute, and failing to tune proxy cost weights to account for a drastically different technology node size.
The Acknowledgements section of Cheng et al. also lists the Nature corresponding authors and implies that they were consulted or even involved, but this is not the case. In fact, the corresponding authors only became aware of this paper after its publication.
Could you make a comparison between FlyCode and Stripe Dunning / stunning.co [1]?
When should a B2B SaaS reach for FlyCode (an app on Stripe Marketplace) vs Stripe Dunning (I'm actually unsure if it's their own offering or not)?
The biggest difference is that we own the retries and communications e2e (transactional email/sms from your domain and dedicated #). This enables us to be far more configurable for each Merchant in terms of recovery period while ensuring that sufficient retry and communications are occurring prior to cancelling a subscription. Stunning is typically used alongside Stripe's retries to send customer emails, which means the two are disjointed. This often leads to too many communications or too few. By treating each customer and their payments individually we're taking a highly tailored approach that's fully automated.
Stripe also caps retries at 8 attempts and while you don't want to over-attempt there are many payments left on the table that require more. It's the card networks (visa, mastercard, etc.) that set retry rules. Unless you're on an IC+ model Stripe is absorbing the declined authorization fees so there's a partial conflict of interest here.
Each business is different in terms of average transaction size, failure rate and recovery rate. Our primary value prop is increasing recovery rate but for others is our automation that's even more valuable to scale operations efficiently and move manual outreach efforts to other areas of the business.
Been working on it on and off for 4 years now. I used to lead an ML consultancy and one thing I've learned is that data scientist DO NOT understand reverse proxies, nvidia drivers, docker and can not set up complex environments for their projects, especially ones that would support collaborative work.
It's been really fun to work on this, currently working on Kubernetes support.