Interesting that the hardware is NVidia Blackwell, not Google TPUs. That means Google will likely have an energy efficiency and cost advantage, and keep their proprietary hardware out of other people's reach.
Getting a whole business set up to build TPU hardware for third parties (design, build, sell, support, etc.) is probably not worth it when there is overflowing demand for TPUs in their cloud already.
Businesses running their own hardware probably prefer CUDA as well for being more generally useful.
Part of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but customers like the portability of targeting CUDA
My limited understanding is that CUDA wins on smaller batches and jobs but TPU wins on larger jobs. It is just easier to use and better at typical small workloads. At some point for bigger ML loads and inference TPU starts making sense.
Not really. Reverse engineering a modern chip is no small feat. Any company capable of it is also capable of designing their own from scratch. However getting something taped out (and debugged) on a modern process is massively expensive.
I did my undergrad internship on federated learning. I was tasked with implementing in a simulator different federated algorithms, so to have a way to compare them in a meaningful way. The last that had to be implemented was FedMA. We didn't manage to do it. That algorithm is absolutely devilish. Every issue that I solved made other two issue arise, and neither my supervisors could help. The sheer idea of matching neurons in different networks might (and does) make sense, but the way the approximate costs are calculated require other 2/3 math papers that I could follow for only the first lines of the abstract. I'm happy for the time I spent in my internship there. I'm also happy it's over
The general understanding of how it works is surprisingly easy though, you can find the paper here https://arxiv.org/abs/2002.06440
That's the point of the privacy scheme. It would only be able to learn things common to multiple clients. Private data wouldn't make it through the noise.
That is beside the point. No lawyer will allow any "learning" on private data. Even without legal counsel, companies would be stupid to do it without compensation.
How is it beside the point? Legal might initially advise against cloud storage but if you come back to them with one that's encrypted on the client side that likely changes things.
Compensation could be of the form "we get a cheaper rate from Google" or even "this is the only form in which the service is offered" or perhaps "we aren't big enough to qualify for the fully airgapped offering".
The point is, for this to work, your system has to be connected to the internet. And if that's the case, there's no material difference between hosting it on prem and hosting it on GCP (where Google can promise you that it won't exfiltrate data). That's if you trust Google. And if you don't (and you shouldn't - it's an ad / mass surveillance company first and foremost), your only option for sensitive data is to self host and air gap.
A bit thin on detail, but will this require confidential VMs with encrypted GPUs? (And I wonder how long before someone cracks SEV-SNP and TDX and pirate copies escape into the wild.)
The Common Crawl is going to become increasingly contaminated with LLM output and training data that is more likely to have less LLM output will become more valuable.
I see this misconception all the time. Filtering out LLM slop is not much different than filtering out human slop. If anything, LLM generated output is of higher quality that a lot of human written text you'd randomly find on the internet. It's no coincidence that state-of-art LLMs increasingly use more and more synthetic data generated by LLMs themselves. So, no, just because training data was produced by a human doesn't make it inherently more valuable; the only thing that matters is the quality of the data, and the Internet is full of garbage which you need to filter out one way or another.
But the signals used to filter out human garbage are not the same the signals that would be needed to filter LLM garbage. LLMs generate texts that look high-quality at a glance, but might be factually inaccurate. For example, an LLM can generate a codebase that is well-formatted, contains docstrings, comments, maybe even tests; but it will use a non-existent library or be logically incorrect.
The number of folks that have the hardware at home to run it is going to be very low and the risk of companies for leaking it is gonna make it unlikely IMHO.
I don't know – if there's still dumb money being thrown towards AI in non-tech and non-privacy-heavy industries, especially ones traditionally targeted by ransomware, there'll always be a chance of datasets getting leaked. I'm thinking retail and consumer product-oriented companies. (There's always non-Western governments without strong security orgs, too.)
Financial firms with significant on-prem datacenter use will love this as well. My company still stays away from the cloud -- we have 6 DCs in the building, and run everything else out of colocated racks.
I don’t think so. To my knowledge GCP has no approval for classified networks, which is by far the hardest part. Contrast with Azure OpenAI has been approved to run on government networks for over a year now.
This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.
>Today at Google Cloud Next, we're thrilled to announce another significant milestone for Google Public Sector: the authorization of Google Distributed Cloud Hosted (GDC Hosted) to host Top Secret and Secret missions for the U.S. Intelligence Community, and Top Secret missions for the Department of Defense (DoD).
> Our GDC air-gapped product, which is now authorized for US Government Secret and Top Secret missions, and on which Gemini is available, provides the highest levels of security and compliance.
Banking as well, this is the kind of offering they've been looking for a while. Google just saw the demand decided to jump in while OpenAI and Anthropic probably calculated they don't have the manpower to deal with the support for this.
With a few exceptions for companies with highly secretive data, you do have to be a government agency or working in a highly regulated government-adjacent area for secured private clouds to be a requirement carved in stone and therefore worth investing a ton of extra money into though.
Neither approaches the secrecy needed by government installations. Health care and banking leak PII regularly and never really suffer any consequences.
i'll add that on-prem is getting 10-100x easier than it was 10-20 years ago (still very hard), and "i want to run this in my own datacenter" is becoming accessible to much smaller companies than just F500 enterprises
Curious if this was forced on Google Cloud by Sundar, or was it something that Google Cloud as an org wanted to do?
At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.
Absolutely many would, especially those with deep pockets. The biggest concern I'm hearing from companies adopting AI, for basically any use case, is data leaving their network. Especially (but not only) in the EU.
I don't understand how Google is willing to do this but won't sell TPUs to other days centers. It should be obvious from Nvidia's market cap that they're missing a huge opportunity.
The only reasons I can think of is they see them as their secret sauce, they don't want to support them for customers long-term, or they don't have the foundry capacity.
It's definitely #3. The GPUs have to first satisfy Google's own computing needs, and only then can they start selling them to others. Given how much training and inference the company is doing and how much demand there is internally it's very unlikely they are able to manufacture loads of extras, especially not profitably.
Would Google seriously have trouble raising the funds to build a chip fab? This seems like something they could do if they actually want to but I’d guess that would take actual leadership when they appear to have none.
Especially in today’s political climate, building this in a purple state would ensure longevity too. The Trump admin would probably let them break ground immediately if they had the plans and I doubt democratic leadership would disagree either.
If I was an investor and Google said they are going to now compete with Nvidia and TSMC I would take that as a sign they the leadership has completely lost the ability to see what their core competency is. Investing 100-200+ billion into fabs just to be on an equal playing field, is not it.
Would be a poor allocation of capital. Especially since, as they build up capacity for their own jobs, they get to see the excess to customers.
That’s the first thing I thought of as well. I had to integrate one into our custom CMS early in my career. I vaguely remember explaining to management that I was not responsible for the order or quality of search results and tweaking queries (now prompts?) with hints to restrict searches to certain paths. It was such an opaque device, but provided better results than MySQL did at the time.
The raw computation is just a bunch of matrix multiplications in a row, most of the algorithmic complexity/secret stuff would be around scaling & efficiency.
For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network.
This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model).
I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).
The model itself is likely built upon their own open source system JAX so they should be usable in Nvidia. Of course cost efficiency is going to be a different story.
Seems pretty high, this is an air gapped product so at some point the employees of whatever government they are giving it to would need to SSH into the VM's to load new weights etc. Lots of ways to make it tricky/watermark the weights though.
The google search appliance might have been one of the worst products I've ever used in my career. If they're going to make a box, I hope they put some effort into it.
Probably the most praise I’ve ever seen about Elastic.
I do respect the amount of power and utility, and it’s definitely a workhorse, but it’s like a horse with one human leg, a bad eye, extra bones but also not enough bones, and a French accent but only knows Korean. Once you get used to the fact that you can’t do what you intend to, but you can do what elastic wants, it becomes a lot more manageable.
I don't think GP is talking about the "search with google" box on third party sites. They're talking about a physical on-prem search server (box) that google used to distribute.
I wonder if the improvements in semantic search have changed that at all. For a big company though, you might need a pretty beefy setup to perform the initial indexing.
The Netflix appliance is pretty good in my experience. No reason Google couldn’t pull something similar off themselves, unless they’re being very Google about it.
Given the myraid of issues they seem to have, I am not sure I would classify Pixels as having polish. But yes, they definitely have the talent to make some good hardware. It's just a matter of whether their priorities match those of their users.
I can't think of a major phone brand that hasn't had some kind of major issue over the years. The batteries of the Note 7, iPhone "antenna gate" (and the more recent lack of advertised AI debacle), etc.
I think Pixels are pretty polished, at least compared to all the cheapo off-brand Android options out there. Some people like Samsung better but I can't stand their UI. Apple would be fine if I could sideload...
They don't sell them. But, if the developer / hotelier had a sufficiently large network, think providing service equivalent to the number of rooms at a US state university system network (multiple universities), then they might qualify: https://openconnect.netflix.com/en/
There are plenty of hotel groups big enough for that, but their properties are geographically distributed and I can't imagine they'd benefit from running fibre for their own multi-site network. Better to just connect each property to a local ISP like everyone else.
Maybe there are some exceptions. Disney World? MGM Resorts in Las Vegas?
FTA:
As part of the announcement, Google said Nvidia
will bring Gemini models to the company’s Blackwell graphics processing units, or GPUs. Companies can buy the chips through Google or other channels.
Richard has a student with an idea involving AI and joins his company as an advisor but can't keep his opinions to himself. Ends up ruining the company because everything he touches turns to shit.
On paper, Stephen Tobolowsky seems like he shouldn’t be successful enough of an actor to warrant an autobiography. But man do I love Ned “The Head” Ryerson in all his incarnations. What a strange, tall, little man.
That the world does not have a Stanley Tucci, Stephen Tobolowsky buddy comedy trilogy has made it all the poorer. But it’s been a while since someone tried to remake The Odd Couple…
I don't think you're wanting to converse in good faith, but on the off chance this is a question - yes, GCP was revenue losing for a number of years, but since Q1 2023 they've been profitable. It takes money to bootstrap anything - obviously - this is the case for the vast majority of companies and their offerings, especially so for one which requires vast amounts of compute resources, SREs, legal, etc.
Advertisers paid money for Google for totally unrelated services. Google invested that money in a number of ways. One of them was to build this very profitable non-advertising business. The advertisers didn't fund that business any more than the advertisers funded US treasuries, or the dozens of startups that Google has invested in as a VC.
This is a thread about using your money for better things than paying an ad company. The comment that started this argument you want to have pointed out that it’s self sustaining. But I pointed out that wasn’t always true. Tfsh backed my claim.
So today maybe there isn’t a problem to which your money isn’t being spent with the ad org but it was that way for a very long time to which we can grant the OP some grace as it’s a rather recent change.
There is even still an argument to be made that while you may not be giving money to the ad org you are still giving money to Google thereby helping them deflect the damage they cause the world in their other orgs.
No, even if you were Google Cloud paying customer #1, your money was going to Cloud. It wasn't supporting anything to do with ads.
The ads were providing income to Google which allowed Google to bootstrap Cloud until it was profitable on its own, not vice-versa.
When you buy (or bought) Cloud services, that doesn't affect Google's ad revenue or advertising behavior at all, not for the better and not for the worse. They're basically unrelated orgs within the corporation. Using Cloud isn't promoting ads or whatever you seem to think, not now and not previously.
But it’s not about killing Google’s ad revenue, it’s about hurting Google as a whole. It’s a complete monster, regardless how many heads the hydra has.
You could have saved us all a lot of time by simply stating upfront that you hate Google as a whole, rather than discussing the technicalities of which parts have to do with advertising or not.
Would you prefer VCs to have fronted the money to bootstrap it? How is it relevant today if ads are no longer enabling their financial viability? Ads largely finance Google's consumer offerings, not their enterprise offerings. Most enterprise Google customers understand the difference.
I believe these are pure word tricks to suggest privacy without actually delivering it.
As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR.
If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.
Why is don't be evil relevant here? If Google never had that motto would you care less? It's not even factual that they dropped it from the code of conduct. It was just moved to the end rather than at the beginning. Moving it wasn't some magical event that signaled a change in Google's ethical values. Do the right thing was just seen as less ambiguous and placed more prominently.
As others have stated, being able to see that the appliance is phoning home or not is trivial. No one who is in the market for this won't ensure it meets some rigurous bar.
You’re talking about Fortune 50 companies here. I don't think Google is going to be messing around spying on them in direct violation of the no-doubt sophisticated contract that will be signed between them.
That was not with Google's consent and it was quickly shut down by enabling encryption between nodes in Google's internal networks. Your average company is far more likely to be susceptible to state actors than Google is.
According to the documents leaked by Edward Snowden, that espionage was sniffed in-transit in plaintext across the Internet's trunk and filtered against XKEYSCORE queries for eventual collection. Google's surprise came from the expectation that cross-datacenter traffic was sent over direct circuits and not susceptible to interception.
It was totally unrelated to PRISM, which was more like a voluntary law enforcement access portal that autoapproved every request. The participating companies since made public statements saying they no longer operate the portal, thereby forcing intelligence agencies to use National Security Letters instead. That's certainly closer to the intent of the laws passed by Congress.
> FISA orders and authorizations can be used to compel electronic surveillance and the disclosure of stored data, including content from services like Gmail, Drive, and Photos.
That's very different from prism. It's also why Google has spent a lot of energy trying to make it impossible for them to see the contents of your data. The government cannot conpel information Google doesn't have access to. I'm also not sure it's relevant for the topic of this post.
You’re making a lot of assumptions there. It’s trivial to monitor traffic patterns from modern appliances, even if it’s encrypted.
Also, companies have been sharing data with cloud security organisations for years now. There a robust means of assessing the risk. License agreements are a very real thing.
I don't fully disagree, but the only reason why this product is noteworthy is precisely because companies don't trust cloud providers with their data anymore. And while you might be able to prevent data exfiltration by monitoring the traffic patterns, you probably can't prevent sabotage that way.
Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business? The legal and reputational damage far outweigh the value of stolen information.
Or do you mean that it could be a vector of attack? That can happen with literally any piece of software, hardware, or appliance you install in or out of your datacentre.
> Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business?
The US government is constantly telling us that the likes of Huawei and Hikvision are doing precisely that, despite being subject to the same risks of reputational damage.
Of course, the same could be said of everything else in the data centre. It's not like Google are somehow more vulnerable than Juniper or Cisco or Unifi or Dell or Intel or whoever.
It's the same folks it always has been. Google is just trying to win those customer's business that would never have otherwise chosen Google. I'm sure these on prem solutions are not nearly as cost efficient as running the same workloads in Google data centers. Most companies would not pay that difference unless forced to via regulatory requirements.
That and there are various regulatory, political etc. reasons. Also I'm not sure about the "anymore" IMHO a lot more companies trust cloud providers with their data than they did 10-20 years ago .
Well, TFA appears to be thin on the details, but who says whatever they deploy is phoning home? If you run their model on prem, it wouldn't be a difficult feat to monitor its network traffic. Not to mention limiting it. It would be tricky if it phoned home by design, but if this is all abstracted through tool use or something, it can certainly be audited. And the kind of company that wants this usually doesn't just run random software without understanding and inspecting closely what it does.
This is being sold as an air gapped product, it has to work offline by definition.
Sure you could hide some way of phoning home and deploy it into the SCIF, but would you really want to risk a firing squad to improve some advertising metrics?
Google announcement:
https://cloud.google.com/blog/products/ai-machine-learning/r...
Interesting that the hardware is NVidia Blackwell, not Google TPUs. That means Google will likely have an energy efficiency and cost advantage, and keep their proprietary hardware out of other people's reach.
Getting a whole business set up to build TPU hardware for third parties (design, build, sell, support, etc.) is probably not worth it when there is overflowing demand for TPUs in their cloud already.
Businesses running their own hardware probably prefer CUDA as well for being more generally useful.
Part of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but customers like the portability of targeting CUDA
Which use cases are TPUs superior for?
Running Gemini models, for one.
What are the pros of using CUDA-enabled devices for inference?
My limited understanding is that CUDA wins on smaller batches and jobs but TPU wins on larger jobs. It is just easier to use and better at typical small workloads. At some point for bigger ML loads and inference TPU starts making sense.
Google doesn’t make TPUs available to 3rd parties, right? I assume there would be tremendous reverse-engineering risk if they were to include them?
Not really. Reverse engineering a modern chip is no small feat. Any company capable of it is also capable of designing their own from scratch. However getting something taped out (and debugged) on a modern process is massively expensive.
> not Google TPUs
They're in limited supply. Even Google doesn't have enough for their own use.
It’s telling that their effort was to get an air gapped solution cleared for US government and the US military.
This might be a great way for them to strengthen their model through federated learning.
https://federated.withgoogle.com/
I did my undergrad internship on federated learning. I was tasked with implementing in a simulator different federated algorithms, so to have a way to compare them in a meaningful way. The last that had to be implemented was FedMA. We didn't manage to do it. That algorithm is absolutely devilish. Every issue that I solved made other two issue arise, and neither my supervisors could help. The sheer idea of matching neurons in different networks might (and does) make sense, but the way the approximate costs are calculated require other 2/3 math papers that I could follow for only the first lines of the abstract. I'm happy for the time I spent in my internship there. I'm also happy it's over
The general understanding of how it works is surprisingly easy though, you can find the paper here https://arxiv.org/abs/2002.06440
The whole point of deploying such things on-prem is air-gapping it from Google and its "learning".
That's the point of the privacy scheme. It would only be able to learn things common to multiple clients. Private data wouldn't make it through the noise.
That is beside the point. No lawyer will allow any "learning" on private data. Even without legal counsel, companies would be stupid to do it without compensation.
How is it beside the point? Legal might initially advise against cloud storage but if you come back to them with one that's encrypted on the client side that likely changes things.
Compensation could be of the form "we get a cheaper rate from Google" or even "this is the only form in which the service is offered" or perhaps "we aren't big enough to qualify for the fully airgapped offering".
The point is, for this to work, your system has to be connected to the internet. And if that's the case, there's no material difference between hosting it on prem and hosting it on GCP (where Google can promise you that it won't exfiltrate data). That's if you trust Google. And if you don't (and you shouldn't - it's an ad / mass surveillance company first and foremost), your only option for sensitive data is to self host and air gap.
A bit thin on detail, but will this require confidential VMs with encrypted GPUs? (And I wonder how long before someone cracks SEV-SNP and TDX and pirate copies escape into the wild.)
At the pace models improve, the advantage of going the dark route shouldn't really hold for long, unless I'm missing something.
Access to proprietary training data: Search, YouTube, Google Books might give some moat.
We have Common Crawl, which is also scraped web data for training LLMs, provided for free by a non-profit.
The Common Crawl is going to become increasingly contaminated with LLM output and training data that is more likely to have less LLM output will become more valuable.
I see this misconception all the time. Filtering out LLM slop is not much different than filtering out human slop. If anything, LLM generated output is of higher quality that a lot of human written text you'd randomly find on the internet. It's no coincidence that state-of-art LLMs increasingly use more and more synthetic data generated by LLMs themselves. So, no, just because training data was produced by a human doesn't make it inherently more valuable; the only thing that matters is the quality of the data, and the Internet is full of garbage which you need to filter out one way or another.
But the signals used to filter out human garbage are not the same the signals that would be needed to filter LLM garbage. LLMs generate texts that look high-quality at a glance, but might be factually inaccurate. For example, an LLM can generate a codebase that is well-formatted, contains docstrings, comments, maybe even tests; but it will use a non-existent library or be logically incorrect.
Problem with filtering is that LLMs can generate few orders of magnitude more slop than humans.
LLM output is uniquely harmful because LLMs trained on LLM output are subject to model collapse
https://www.nature.com/articles/s41586-024-07566-y
Are the differences between Google Books and LibGen documented anywhere? I believe most models outside of Google are trained on the latter.
The number of folks that have the hardware at home to run it is going to be very low and the risk of companies for leaking it is gonna make it unlikely IMHO.
I think home users would be the least of their concerns.
It only takes one company to leak it
Or one company to get hacked and the hackers leak it
Realistically the only people able to run models of this size are large enterprises.
Those enterprises won’t take the risk of being sued for using a model without proper permission.
I don't know – if there's still dumb money being thrown towards AI in non-tech and non-privacy-heavy industries, especially ones traditionally targeted by ransomware, there'll always be a chance of datasets getting leaked. I'm thinking retail and consumer product-oriented companies. (There's always non-Western governments without strong security orgs, too.)
Nations.
or large government sponsored entities like Mossad. Air gapping won't protect against spying. Good luck trying to sue them
They can get "hacked" and wooops.
> I wonder how long before someone cracks SEV-SNP
https://bughunters.google.com/blog/5424842357473280/zen-and-...
I'd expect watermarked model weights plus a lot of liability to distinctivise leaking the model.
This is obvious government contract baiting. Kudos though they might actually move some Google Distributed Cloud this way
Financial firms with significant on-prem datacenter use will love this as well. My company still stays away from the cloud -- we have 6 DCs in the building, and run everything else out of colocated racks.
Who provides internet
> financial firm
That means they have direct fiber connections to the Tier 1/Tier 2 guys. The big ones have direct fiber connections to the NYSE.
If they're not rolling their own connection, they're using BT Radianz, IPC Systems, Colt Technology Services, etc.
I don’t think so. To my knowledge GCP has no approval for classified networks, which is by far the hardest part. Contrast with Azure OpenAI has been approved to run on government networks for over a year now.
This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.
>Today at Google Cloud Next, we're thrilled to announce another significant milestone for Google Public Sector: the authorization of Google Distributed Cloud Hosted (GDC Hosted) to host Top Secret and Secret missions for the U.S. Intelligence Community, and Top Secret missions for the Department of Defense (DoD).
https://cloud.google.com/blog/topics/public-sector/google-pu...
You are right, I should’ve RTFA
FedRAMP High is the mark you really want to hit for the US Government and GCP's service coverage is surprisingly broad in that realm.
From Google's blog post:
> Our GDC air-gapped product, which is now authorized for US Government Secret and Top Secret missions, and on which Gemini is available, provides the highest levels of security and compliance.
Banking as well, this is the kind of offering they've been looking for a while. Google just saw the demand decided to jump in while OpenAI and Anthropic probably calculated they don't have the manpower to deal with the support for this.
This is obvious government contract baiting
You don't have to be a government agency to not want your company's data all over the place.
With a few exceptions for companies with highly secretive data, you do have to be a government agency or working in a highly regulated government-adjacent area for secured private clouds to be a requirement carved in stone and therefore worth investing a ton of extra money into though.
Just off the top of my head: Healthcare. Banking.
Neither approaches the secrecy needed by government installations. Health care and banking leak PII regularly and never really suffer any consequences.
This is just inaccurate
https://circlesofcaredatasettlement.com/
Anecdotes are not data.
Healthcare and banking have no issue storing data in third party datacenters as long as they meet the applicable standards.
You absolutely don't need this or FedRAMP to do healthcare.
i'll add that on-prem is getting 10-100x easier than it was 10-20 years ago (still very hard), and "i want to run this in my own datacenter" is becoming accessible to much smaller companies than just F500 enterprises
Huh? 20 years ago on-prem was the norm and the cloud didn’t exist.
Yeah, AWS was in private beta in 2005 and public in 2006. People knew it existed but didn't have access yet. https://www.zdnet.com/article/how-amazon-exposed-its-guts-th...
Both of which are encumbered with regulations that want them to need this.
They’ll have to fight Microsoft who’s been promising copilot.
Google AI rackmount appliance coming soon?
Curious if this was forced on Google Cloud by Sundar, or was it something that Google Cloud as an org wanted to do?
At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.
I assume Google Distributed Cloud is part of the larger Cloud org so they get the revenue either way. The on-prem version may even cost more.
Like you can with deep seek? Or will it be more complicated and expensive. I don't know who would actually want that.
Absolutely many would, especially those with deep pockets. The biggest concern I'm hearing from companies adopting AI, for basically any use case, is data leaving their network. Especially (but not only) in the EU.
Deepseek is just the model weights. Nothing about it requires network access.
Deepseek is not really comparable to Gemini 2.5 Pro.
DeepSeek-R2 may be...
Folks who would prefer to run deepseek are not in the end customer for this product. Deepseek doesn't provide a service contract.
I work for a bank and we're banned from using AI because of the privacy problem. I'd think there is a big market for a product like this.
I don't understand how Google is willing to do this but won't sell TPUs to other days centers. It should be obvious from Nvidia's market cap that they're missing a huge opportunity.
The only reasons I can think of is they see them as their secret sauce, they don't want to support them for customers long-term, or they don't have the foundry capacity.
It's definitely #3. The GPUs have to first satisfy Google's own computing needs, and only then can they start selling them to others. Given how much training and inference the company is doing and how much demand there is internally it's very unlikely they are able to manufacture loads of extras, especially not profitably.
This is a trillion dollar corporation.
Would Google seriously have trouble raising the funds to build a chip fab? This seems like something they could do if they actually want to but I’d guess that would take actual leadership when they appear to have none.
Especially in today’s political climate, building this in a purple state would ensure longevity too. The Trump admin would probably let them break ground immediately if they had the plans and I doubt democratic leadership would disagree either.
So what gives?
Building a fab is pretty much impossible. You can give TSMC a ton of money to build more capacity, but so can everyone else in the space.
TPU's are vertically integrated.
If I was an investor and Google said they are going to now compete with Nvidia and TSMC I would take that as a sign they the leadership has completely lost the ability to see what their core competency is. Investing 100-200+ billion into fabs just to be on an equal playing field, is not it.
Would be a poor allocation of capital. Especially since, as they build up capacity for their own jobs, they get to see the excess to customers.
Reminds me of the Google search appliance.
That’s the first thing I thought of as well. I had to integrate one into our custom CMS early in my career. I vaguely remember explaining to management that I was not responsible for the order or quality of search results and tweaking queries (now prompts?) with hints to restrict searches to certain paths. It was such an opaque device, but provided better results than MySQL did at the time.
Is Gemini tied/benefitting from Google TPU hardware? Because you need hardware in the data center to run this, and I feel it is somewhat specialised.
Gemini models are written in Jax which through the XLA compiler can be compiled either to TPU or GPU hardware.
Performance may differ but Google (and Nvidia) are very interested in having good performance on both platforms.
The raw computation is just a bunch of matrix multiplications in a row, most of the algorithmic complexity/secret stuff would be around scaling & efficiency.
For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network.
This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model).
I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).
The model itself is likely built upon their own open source system JAX so they should be usable in Nvidia. Of course cost efficiency is going to be a different story.
TPUs are definitely the reason why Gemini models have both massive context and very low prices. There is no nvidia tax to pay.
The Google blogpost notes that it's a partnership with Nvidia, so using cuda rather than TPUs apparently.
Makes complete sense, as NVidia has a lot more experience building these types of appliances.
Some one said it could also mean Google hardware has some advantage they would rather stay inside the G-silo.
Google abandoned Coral in true Google style.
What is the risk that some hacker could exfliltrate the weights?
Seems pretty high, this is an air gapped product so at some point the employees of whatever government they are giving it to would need to SSH into the VM's to load new weights etc. Lots of ways to make it tricky/watermark the weights though.
Very low if they use confidential VMs (CPU rooted encryption). Just like the Xbox uses and remains unhacked 10 years later.
What would be the benefit of hacking the Xbox? What would you get?
Pirated games, cheating/botting in multiplayer rooms, etc.
And fame.
What a sudden change of heart. Thank you, DeepSeek!
Not a cat's chance in hell that any eu organisation will rush to this offer right now. Or maybe ever in fact.
Huh, how does that matter? Maybe OVH AI is fine for EU.
Is The Gavin Belson Signature Edition Box is needed to run these?
The google search appliance might have been one of the worst products I've ever used in my career. If they're going to make a box, I hope they put some effort into it.
I actually mourn the loss of it. It felt so much better than any other accursed on-prem search solution I've seen since.
fastsearch was pretty good before Microsoft bought it.. Elastic is good enough.
> Elastic is good enough.
Probably the most praise I’ve ever seen about Elastic.
I do respect the amount of power and utility, and it’s definitely a workhorse, but it’s like a horse with one human leg, a bad eye, extra bones but also not enough bones, and a French accent but only knows Korean. Once you get used to the fact that you can’t do what you intend to, but you can do what elastic wants, it becomes a lot more manageable.
The box might have been awful, but the sigh of relief when seeing a website was using it was wonderful.
I don't think GP is talking about the "search with google" box on third party sites. They're talking about a physical on-prem search server (box) that google used to distribute.
Strictly speaking Google still make boxes for people just in a different market.
What was so bad about the search appliance though? Physical? Software?
My theory is that the heuristics (PageRank and click-through feedback) that made 2008 Google great don't work in corporate environments.
Hiring a 2010-era SEO expert to spam the company Notion with backlinks to the stuff I've written.
Common Confluence SEO at my company is to add a bunch of keywords then change the text color to white!
I wonder if the improvements in semantic search have changed that at all. For a big company though, you might need a pretty beefy setup to perform the initial indexing.
The Netflix appliance is pretty good in my experience. No reason Google couldn’t pull something similar off themselves, unless they’re being very Google about it.
They have some incredible hardware talent (TPUs, Pixels), but I'm guessing this project will not get the polish of those more public facing products
Given the myraid of issues they seem to have, I am not sure I would classify Pixels as having polish. But yes, they definitely have the talent to make some good hardware. It's just a matter of whether their priorities match those of their users.
I can't think of a major phone brand that hasn't had some kind of major issue over the years. The batteries of the Note 7, iPhone "antenna gate" (and the more recent lack of advertised AI debacle), etc.
How would you say it compares to those?
I think Pixels are pretty polished, at least compared to all the cheapo off-brand Android options out there. Some people like Samsung better but I can't stand their UI. Apple would be fine if I could sideload...
Samsung with a 3rd party launcher works pretty well
Do they only sell those to ISPs or could a housing developer or a hotelier get their hands on those?
They don't sell them. But, if the developer / hotelier had a sufficiently large network, think providing service equivalent to the number of rooms at a US state university system network (multiple universities), then they might qualify: https://openconnect.netflix.com/en/
There are plenty of hotel groups big enough for that, but their properties are geographically distributed and I can't imagine they'd benefit from running fibre for their own multi-site network. Better to just connect each property to a local ISP like everyone else.
Maybe there are some exceptions. Disney World? MGM Resorts in Las Vegas?
Why? In my limited experience, it was pretty useful, but again, my experience is limited.
What made it one of the worst products you’ve ever used?
FTA: As part of the announcement, Google said Nvidia will bring Gemini models to the company’s Blackwell graphics processing units, or GPUs. Companies can buy the chips through Google or other channels.
More like the GSA probably.
https://en.m.wikipedia.org/wiki/Google_Search_Appliance
which is a black-box more or less, so the "own data center" is a bit vague as concept here.
It's quite clearly a yellow-box, made from The Chest hide.
https://homestarfanstuff.fandom.com/wiki/The_Cheat
right, when you return the box. Google can then retrieve all the logs.
No, Jack Barkers' revamped version is needed.
You're going to need a few specialized racks all wired up together. A single box won't be sufficient.
He personally stands behind the developed haedware, will you stand with him ?
I got that reference
I miss that show. Too bad it ended right before the AI hype.
Fingers crossed that it'll do an Arrested Development.
Richard has a student with an idea involving AI and joins his company as an advisor but can't keep his opinions to himself. Ends up ruining the company because everything he touches turns to shit.
Ah, I’d forgotten how much I wanted to strangle Richard by the end of that show. He kind of lived long enough to become the villain.
That was the point of his character arc. All SV Big tech Bros turn into the people they sought to destroy. Chasing money and growth.
Wasn't the last season mainly about AI?
In fact, an AI that went rogue was the major plotpoint, so the satire is still on point.
I miss it too.
I think by the end I was far more invested in the characters rather than the plot though.
> Too bad it ended right before the AI hype.
It was the very start of the AI hype cycle, and in fact they built the app: https://news.ycombinator.com/item?id=14636228
This tactic comes straight out of the Conjoined Triangles of Success playbook. It’s a classic Action Jack Barker move.
On paper, Stephen Tobolowsky seems like he shouldn’t be successful enough of an actor to warrant an autobiography. But man do I love Ned “The Head” Ryerson in all his incarnations. What a strange, tall, little man.
That the world does not have a Stanley Tucci, Stephen Tobolowsky buddy comedy trilogy has made it all the poorer. But it’s been a while since someone tried to remake The Odd Couple…
His role in Californication always makes me laugh
His signature could be the size of a full size rack on this one.
[dead]
It's still an advertising company you're doing business with.
I mean, would you buy cookies from a brand that is known for producing rodenticides?
I buy Lidl store brand biscuits as well as Lidl store brand cleaning products, among many other things, so I guess I would.
Lidl produces none of those, just brands them. More comparable would be something like Raid cookies I guess
Didn't google just buy deepmind and rebrand it too?
Lidl produces a lot of products on their own, as well as packaging (Schwarz Produktion) [0] https://en.m.wikipedia.org/wiki/Schwarz_Group
Google Cloud had $50B in revenue last year, so clearly plenty of companies didn't get your memo.
...yes?
I mean, the company that makes Raid also makes Saran Wrap and Ziploc bags. Corporate conglomerates can do lots of things.
The entire Google Cloud org is funded by regular customers paying money, not advertising.
> The entire Google Cloud org is funded by regular customers paying money
That can’t be true, how did they bootstrap it? How do they pay for R & D for their half baked offerings?
I don't think you're wanting to converse in good faith, but on the off chance this is a question - yes, GCP was revenue losing for a number of years, but since Q1 2023 they've been profitable. It takes money to bootstrap anything - obviously - this is the case for the vast majority of companies and their offerings, especially so for one which requires vast amounts of compute resources, SREs, legal, etc.
So just to clarify the entire cloud org was funded by advertisers for most of it’s existence.
No, it was funded by Google.
Advertisers paid money for Google for totally unrelated services. Google invested that money in a number of ways. One of them was to build this very profitable non-advertising business. The advertisers didn't fund that business any more than the advertisers funded US treasuries, or the dozens of startups that Google has invested in as a VC.
What’s the problem? Google is trying to diversify their revenue streams. I don’t understand the relevance. Apple TV+ is paid for by iPhones. Ok? And?
> Ok? And?
This is a thread about using your money for better things than paying an ad company. The comment that started this argument you want to have pointed out that it’s self sustaining. But I pointed out that wasn’t always true. Tfsh backed my claim.
So today maybe there isn’t a problem to which your money isn’t being spent with the ad org but it was that way for a very long time to which we can grant the OP some grace as it’s a rather recent change.
There is even still an argument to be made that while you may not be giving money to the ad org you are still giving money to Google thereby helping them deflect the damage they cause the world in their other orgs.
No, even if you were Google Cloud paying customer #1, your money was going to Cloud. It wasn't supporting anything to do with ads.
The ads were providing income to Google which allowed Google to bootstrap Cloud until it was profitable on its own, not vice-versa.
When you buy (or bought) Cloud services, that doesn't affect Google's ad revenue or advertising behavior at all, not for the better and not for the worse. They're basically unrelated orgs within the corporation. Using Cloud isn't promoting ads or whatever you seem to think, not now and not previously.
But it’s not about killing Google’s ad revenue, it’s about hurting Google as a whole. It’s a complete monster, regardless how many heads the hydra has.
OK, well at least you're being honest now.
You could have saved us all a lot of time by simply stating upfront that you hate Google as a whole, rather than discussing the technicalities of which parts have to do with advertising or not.
Would you prefer VCs to have fronted the money to bootstrap it? How is it relevant today if ads are no longer enabling their financial viability? Ads largely finance Google's consumer offerings, not their enterprise offerings. Most enterprise Google customers understand the difference.
Didn’t they start it for themselves and then saw an opportunity to make it a business?
In a sense, yes, it was bootstrapped by ads and now pays for itself.
I believe these are pure word tricks to suggest privacy without actually delivering it.
As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR.
If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.
Why is don't be evil relevant here? If Google never had that motto would you care less? It's not even factual that they dropped it from the code of conduct. It was just moved to the end rather than at the beginning. Moving it wasn't some magical event that signaled a change in Google's ethical values. Do the right thing was just seen as less ambiguous and placed more prominently.
As others have stated, being able to see that the appliance is phoning home or not is trivial. No one who is in the market for this won't ensure it meets some rigurous bar.
You’re talking about Fortune 50 companies here. I don't think Google is going to be messing around spying on them in direct violation of the no-doubt sophisticated contract that will be signed between them.
The NSA has been spying through Google since at least 2009. https://en.wikipedia.org/wiki/PRISM
That was not with Google's consent and it was quickly shut down by enabling encryption between nodes in Google's internal networks. Your average company is far more likely to be susceptible to state actors than Google is.
According to the documents leaked by Edward Snowden, that espionage was sniffed in-transit in plaintext across the Internet's trunk and filtered against XKEYSCORE queries for eventual collection. Google's surprise came from the expectation that cross-datacenter traffic was sent over direct circuits and not susceptible to interception.
It was totally unrelated to PRISM, which was more like a voluntary law enforcement access portal that autoapproved every request. The participating companies since made public statements saying they no longer operate the portal, thereby forcing intelligence agencies to use National Security Letters instead. That's certainly closer to the intent of the laws passed by Congress.
> That was not with Google's consent
The NSA does not need consent from Google. Google is simply ordered to comply. See https://policies.google.com/terms/information-requests?hl=en...
> FISA orders and authorizations can be used to compel electronic surveillance and the disclosure of stored data, including content from services like Gmail, Drive, and Photos.
If you look at the content requests under FISA, you can see that there were over 118000 requests for user data between July 2023 and December 2023. https://transparencyreport.google.com/user-data/us-national-...
That's very different from prism. It's also why Google has spent a lot of energy trying to make it impossible for them to see the contents of your data. The government cannot conpel information Google doesn't have access to. I'm also not sure it's relevant for the topic of this post.
That law applies to all of Google's US customers too.
You’re making a lot of assumptions there. It’s trivial to monitor traffic patterns from modern appliances, even if it’s encrypted.
Also, companies have been sharing data with cloud security organisations for years now. There a robust means of assessing the risk. License agreements are a very real thing.
I don't fully disagree, but the only reason why this product is noteworthy is precisely because companies don't trust cloud providers with their data anymore. And while you might be able to prevent data exfiltration by monitoring the traffic patterns, you probably can't prevent sabotage that way.
I’m confused and not sure what you mean.
Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business? The legal and reputational damage far outweigh the value of stolen information.
Or do you mean that it could be a vector of attack? That can happen with literally any piece of software, hardware, or appliance you install in or out of your datacentre.
> Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business?
The US government is constantly telling us that the likes of Huawei and Hikvision are doing precisely that, despite being subject to the same risks of reputational damage.
Of course, the same could be said of everything else in the data centre. It's not like Google are somehow more vulnerable than Juniper or Cisco or Unifi or Dell or Intel or whoever.
It's the same folks it always has been. Google is just trying to win those customer's business that would never have otherwise chosen Google. I'm sure these on prem solutions are not nearly as cost efficient as running the same workloads in Google data centers. Most companies would not pay that difference unless forced to via regulatory requirements.
> don't trust
That and there are various regulatory, political etc. reasons. Also I'm not sure about the "anymore" IMHO a lot more companies trust cloud providers with their data than they did 10-20 years ago .
I would have more problems with it even being on the network before we start talking about exfiltration.
The boxes can run airgapped: https://cloud.google.com/blog/products/ai-machine-learning/r...
Well, TFA appears to be thin on the details, but who says whatever they deploy is phoning home? If you run their model on prem, it wouldn't be a difficult feat to monitor its network traffic. Not to mention limiting it. It would be tricky if it phoned home by design, but if this is all abstracted through tool use or something, it can certainly be audited. And the kind of company that wants this usually doesn't just run random software without understanding and inspecting closely what it does.
This is being sold as an air gapped product, it has to work offline by definition.
Sure you could hide some way of phoning home and deploy it into the SCIF, but would you really want to risk a firing squad to improve some advertising metrics?