Connect with us

Tech

Microsoft Copilot can now read your screen, think deeply, and speak aloud to you

A week after announcing a wave of updates for its enterprise suite of Copilot AI-powered products, Microsoft is launching new Copilot capabilities on Windows for all users, including a tool that can understand and respond to questions about what’s on your screen.

Refreshed Copilot apps for iOS, Android, Windows and the web are rolling out today, and all feature a Copilot with a more “warm” and “distinct” style, as Microsoft describes it. Microsoft is also bringing the chatbot to WhatsApp, letting users chat with Copilot via DM, similar to the experience you get with other bots on Meta’s messaging platform.

Copilot Vision

Copilot Vision has a view of what you’re viewing on your PC — more specifically, a lens into the sites you’re visiting with Microsoft Edge. Gated behind Copilot Labs, a new, Copilot Pro-exclusive opt-in program for experimental Copilot capabilities, Copilot Vision can analyze text and images on webpages and answer queries (e.g., “What’s the recipe for the food in this picture?”) about them.

Vision, which can be pulled up by typing “@copilot” in Edge’s address bar, isn’t exactly a technical marvel. Google offers similar search technology on Android, and recently brought bits and pieces of that tech to Chrome as well.

But Microsoft suggests that Copilot Vision is more powerful and conscious of privacy than previous screen-analyzing features.

“Copilot Vision can … suggest next steps, answer questions, help navigate whatever it is you want to do, and assist with tasks, all while you simply speak to it in natural language,” Microsoft wrote in a blog post shared with TechCrunch. “Imagine you’re trying to furnish a new apartment. Copilot Vision can help you search for furniture, find the right color palette, think through your options on everything from rugs to throws, and even suggest ways of arranging what you’re looking at.”

Copilot Vision
Using Copilot Vision to ask questions about a photo on the web.
Image Credits: Microsoft

No doubt eager to avoid another round of bad press from AI privacy fumbles, Microsoft is stressing that Copilot Vision was designed to delete data immediately following conversations. Processed audio, images or text aren’t stored or used to train models, the company claims — at least not in this preview version.

Copilot Vision is also limited in the types of websites that it can interpret. For the time being, Microsoft’s blocking the feature from working on paywalled and “sensitive” content, limiting Vision to a pre-approved list of “popular” web properties.

What does “sensitive” content entail, exactly? Porn? Violence? At this juncture, Microsoft wouldn’t say.

Accusations of circumventing paywalls with AI tools have landed Microsoft in legal hot water in the recent past. In an ongoing lawsuit, The New York Times alleged that Microsoft allowed users to get around its paywall by serving NY Times articles through the Copilot chatbot on Bing. When prompted in a certain way, Copilot — which is powered by close Microsoft collaborator OpenAI’s models — would give verbatim (or close-to-verbatim) snippets of paid stories, according to The Times.

Microsoft said that Copilot Vision, which is U.S.-only at the moment, will respect sites’ “machine-readable controls on AI” — like rules that disallow bots from scraping data for AI training. But the company hasn’t said precisely which controls Vision will respect; there are several in use. We’ve asked Microsoft for clarification.

Many major publishers have opted to block AI tools from trawling their websites not only out of fear their data will be used without permission, but also to prevent these tools from sending their server costs soaring. If the current trend holds, Copilot Vision may not work on some of the web’s top news sites.

Microsoft said it’s committed to “taking feedback” to allay concerns.

“Before we launch broadly, we’ll continue to … refine our safety measures and keep privacy and responsibility at the center of everything we do,” Microsoft said in the blog post. “There is no specific processing of the content of a website you are browsing [with Copilot], nor any AI training — Copilot Vision simply reads and interprets the images and text it sees on the page for the first time along with you.”

Think Deeper

As with Vision, Copilot’s new Think Deeper feature is an attempt to make Microsoft’s assistant more versatile.

Think Deeper gives Copilot the ability to reason through more complex problems, Microsoft said, thanks to “reasoning models” that take more time before responding with step-by-step answers.

Which reasoning models? Microsoft was a bit cagey when I asked, saying only that Think Deeper uses “the latest models from OpenAI, fine-tuned by Microsoft.” Reading between the lines, it’s a safe bet that they’re a customized version of OpenAI’s o1 model.

“We’ve designed Think Deeper to be helpful for all kinds of practical, everyday challenges, like comparing two complex options side by side,” Microsoft wrote in a blog post. “Think Deeper can help with anything from solving tough math problems to weighing up the costs of managing home projects.”

Microsoft talked up Think Deeper’s potential quite a bit in its press materials. But assuming the model underneath is o1, it will most certainly fall short in some areas. We’re curious to see what sort of enhancements Microsoft made to the base model, and how forthcoming Think Deeper is about its limitations.

Think Deeper will be available from today to a limited number of Copilot Labs users in Australia, Canada, New Zealand, the U.S. and the U.K.

Copilot Voice

A new Copilot feature generally available today is Copilot Voice (not to be confused with GitHub’s Copilot Voice). Launching in English in New Zealand, Canada, Australia, the U.K. and the U.S. to start, Voice adds four synthetic voices, letting you talk to Copilot and have its responses be spoken aloud.

Copilot Voice
Image Credits: Microsoft

Like OpenAI’s Advanced Voice Mode for ChatGPT, Copilot Voice can pick up on your tone during conversations and respond accordingly, and you can interject at any point while Copilot Voice is answering. A Microsoft spokesperson told me that the mode uses “the latest voice technology with new models that have been fine-tuned for the Copilot app.” What tech? Which models? On the specifics, mum’s the word.

One thing to be aware of: Copilot Voice has a time-based usage limit. Copilot Pro subscribers get more minutes but the number is “variable,” Microsoft told me, depending on demand.

Personalization

Copilot will soon become more tailored to your likes and preferences, Microsoft said, thanks to a new personalization setting.

When the setting is enabled, Copilot will draw on your past interactions and history, as well as your interactions with other Microsoft apps and services (Microsoft won’t say which) to recommend ways to use Copilot.

“This helps you get going,” Microsoft wrote in a blog post, “offering both a handy guide to Copilot’s useful features and conversation starters.”

Personalization in Copilot, which can be switched off in the Copilot settings menu on Windows, isn’t slated for the U.K. or EU anytime soon. But users elsewhere should begin to see the setting this afternoon.

Microsoft and the EU have had a testy relationship where it concerns the company’s AI product rollouts. In May, the EU warned Microsoft that it could be fined up to 1% of its global annual turnover under the bloc’s online governance regime, the Digital Services Act, after the company failed to respond to a request for information that focused on its generative AI tools.

A number of tech giants beyond Microsoft, including Apple and Meta, have taken a cautious approach to launching AI tools in the EU, wary of running afoul of the bloc’s laws governing data privacy and model deployment.

“For users in the European Economic Area (EEA) and a limited number of other countries, we are evaluating options before offering this level of Copilot personalization for those users,” a Microsoft spokesperson told TechCrunch. “Some features will not be available in the EEA until a later date.”

source

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Tech

Can an Apple lawsuit derail OpenAI’s hardware plans?

Apple recently filed a trade secrets lawsuit against OpenAI, accusing the AI company of a pattern of misconduct aimed at getting current and former Apple employees to share confidential information. (In response, OpenAI said it is “not aware of any evidence that this complaint has merit.”)

On the latest episode of TechCrunch’s Equity podcast, Kirsten Korosec, Sean O’Kane, and I debated whether this lawsuit will cast a shadow over OpenAI’s much-discussed plans to get into the hardware business (starting with a mobile smart speaker) and go public.

“Even setting aside whether or not the court grants any kind of injunctive relief or any kind of restraining order over what OpenAI is doing, it just naturally can lead to that sort of situation where it’s going to cause some delays in what OpenAI is working on,” Sean suggested. “Which I’m sure was probably part of the reasoning behind Apple doing this. They don’t do this stuff willy nilly.”

With all those plans on the line, will OpenAI try to settle this as quickly as possible, or did it learn from its recent courtroom victory against Elon Musk that it can endure the cost and embarrassment of a trial? Kirsten, at least, predicts the latter.

Keep reading for a preview of our conversation, edited for length and clarity.

Kirsten: Sean, how do you feel about Sam Altman listening to you with a little device maybe in your pocket?

Sean: I’m good. Maybe that’s predictable, but I’m good. No thanks.

We’ll get into it, I’m sure, but this is allegedly the first product that OpenAI has been working on in its hardware division with Jony Ive and company. They’ve been really coy ever since that weird video they put out last year of them sitting at that coffee shop or bar in San Francisco and sort of talking very vaguely about hardware and legacy devices, meaning laptops and phones. And so if this is the direction they’re headed in, all power to people who want to have somebody like that always listening to them. This is not going to be for me.

Anthony: Part of what we have to remember about those kinds of devices is also that, depending on how mobile it is, it’s not just listening to you, it’s listening to the people around you. I might be fine with it — I’m not fine with it, but let’s say I was — but then if we met up in-person at Disrupt, then suddenly it might be listening to all of us. 

There are all kinds of social norms that are going to have to be renegotiated if these things become widespread. I think we should make fun of and criticize people who record other people without consent.

Kirsten: Well, I bring up the device that has been speculated about for a really long time, and we’ll see what it really ends up being once it’s officially introduced, but it’s important in the context of this lawsuit that Apple filed last Friday. 

It was the biggest news of the week, certainly, and this is a trade secret lawsuit. It has some pretty wild allegations and we should very much emphasize these are allegations that have been filed in a complaint by Apple. But what it is accusing OpenAI of is a pattern of misconduct at the highest levels, specifically directed towards OpenAI employees who used to work at Apple. And in fact they’ve named the chief hardware officer Tang Tan in this lawsuit.

This is all important because Apple is accusing OpenAI of essentially stealing their trade secrets, but in the context of that, this could be then used for a competing hardware product. I’m wondering if maybe we don’t get into whether this lawsuit has merits, because we haven’t gone through full discovery, but what are your initial impressions of the lawsuit aside from the fact that wow, this is going to be entertaining?

Sean: Two things. One, this is a pretty big risk potentially to whatever it is OpenAI is working on. Even setting aside whether or not the court grants any kind of injunctive relief or any kind of restraining order over what OpenAI is doing, it just naturally can lead to that sort of situation where it’s going to cause some delays in what OpenAI is working on, which I’m sure was probably part of the reasoning behind Apple doing this. They don’t do this stuff willy nilly.

The other is that we think that OpenAI is — we know that they’ve filed confidentially for an IPO. We think it might happen as early as the end of this year, or early next year, if you believe Sam Altman’s cautious language around the IPO. And this just raises a whole bunch of questions around that because, on the one hand, we think their business right now is probably overwhelmingly the software; they’re not really factoring in any hardware business into that picture at the moment.

They’re about to go to the markets and they’re going to be pitching bankers and investors on where they think their addressable market should be, and if they have a big amount of that pegged to a potential hardware division and hardware products, this could be a huge risk to that and changes a lot of the calculus of sort of how the IPO gets priced. So that’s where my head’s at.

Anthony: One [allegation] that I assume that Apple must have pretty solid numbers on is, they said more than 400 Apple employees now work at OpenAI. Granted, both of them are very large companies with many thousands or tens of thousands of employees. So as a percentage, it’s not necessarily huge. But that seems like a lot of people and a pretty serious talent drain. 

And the other thing I’m wondering is related to Sean’s point. With the context of the potential IPO, how much damage did OpenAI ultimately take from a marketing and brand perspective from the trial it already went through? That it seemed to basically win, but there was a lot of not-terrible-but-kind-of-embarrassing dirty laundry that came out in the testimony. To what extent are they just like, “We do not want to go through that again”? Or did they take the lesson of, “Hey, we went through it and we survived and we’ll be okay if we have to do another trial with Apple”?

Kirsten: I fully predict the latter, by the way.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

source

Continue Reading

Tech

What to watch for after Jensen Huang’s Japan visit

Nvidia’s chief Jensen Huang spent two days — July 15 and 16 — in Tokyo, courting Japan’s industrial and chip-supply elite, weeks after a keynote in Taiwan, and months after a visit to South Korea. He left with deals spanning Japan’s entire tech ecosystem: a national AI factory, partnerships with the country’s leading robotics companies, and agreements with the chip-material suppliers powering Nvidia’s next generation of AI chips. His message was clear. Nvidia is targeting Japan’s factory floor, and many of the country’s biggest manufacturers are joining in. AI’s next chapter, Huang said, belongs to factory floors, robots, and machines, and he wants Japan to build it.

Thirty years ago, a $5 million Sega investment helped keep a near-bankrupt Nvidia afloat; today, Nvidia and Japan’s industrial giants need each other again — this time to build the physical-AI era, starting with these three projects:

Noetra — Japan’s sovereign-AI play. The country doesn’t want to run its factories and robots on American or Chinese AI. So, the government pulled together roughly 44 domestic firms, with SoftBank, Sony, NEC, and Honda at the core, to build its own AI for robots, vehicles, and factory floors. Tokyo is committing up to 1 trillion yen ($6.2 billion) over five years, a bet on homegrown “physical AI,”  foundation models built to run machines. Japan wants to own the software brain. The hardware to build it, though, still comes from Nvidia. The U.S. chip giant is building “a Vera Rubin AI factory,” a massive data center packed with its next-generation chips, expected to launch in 2028, with 13,750 Vera CPUs and 27,500 Rubin GPUs, delivering 140 megawatts. Noetra will oversee the effort, with plans to build the data center. Noetra’s plan runs in three stages: a reasoning model heavy on Japanese-language skills starting in fiscal 2026; an omni-modal version handling text, images, video, and audio by 2028; and “Real-world Native AI” built to run robots by 2030, released to outside Noetra developers in phases.

The robotics coalition  — Japan’s industrial giants line up behind Cosmos. Nvidia is targeting Japan’s factory floor, and many of the country’s top robotics and manufacturing players are signing on. Fanuc, Yaskawa, Kawasaki Heavy, Fujitsu, Hitachi, NEC, Sony, SoftBank, Kubota, and robotics group AIRoA say they plan to build on Nvidia’s Cosmos models, an open-model effort Nvidia started in May with a handful of global AI labs. In Tokyo, Nvidia gave them a reason to commit, unveiling Cosmos 3 Edge, a version of the model that runs on its Jetson Thor chips inside the machines themselves. Some are already testing a shared control system; others, like Honda R&D and Omron, are building on the tools now. “The next frontier of AI is in the physical world, and this is a once-in-a-generation opportunity for Japan,” Huang said in the company’s statement. “Japan invented modern manufacturing. Now, it has the opportunity to reinvent it for the age of intelligent industries.”

Toyota — cars and physical AI. Toyota uses Nvidia chips across much of its stack. It committed its next-generation vehicles to Nvidia’s Drive platform at CES in January 2025; the newer work extends Nvidia into its manufacturing, where simulations are used to design production lines, into the software that runs its vehicles, and into systems that read road traffic. Toyota’s cars will run advanced driver assistance, which steers and brakes but still requires a driver, a more conservative approach than Waymo and Tesla, which are developing systems that rely less on a human driver.

Huang’s visit put physical AI at the center of Japan’s industrial strategy, and Tokyo is spending to back it. Facing a shrinking workforce, Japan wants 10 million AI-equipped robots across 18 sectors by 2040, backed by $65 billion in public and private physical-AI investment.  

The longer game is bigger. Japan’s AI Robotics Strategy, released in March, aims to capture more than 30% of the global AI robotics market by 2040, a market Tokyo values at roughly ¥20 trillion, or about $133 billion. METI is funding a domestic foundation model to run the machines, and Noetra’s Nvidia-powered factory is where models of that scale, into the trillions of parameters, would be trained.

Underneath the industrial case is a sovereign one. As the U.S. and China pull ahead in large-scale AI, Tokyo wants its own data, its own compute, and less dependence on infrastructure it doesn’t control. Huang appeared on July 16 alongside trade minister Ryosei Akazawa at the government’s physical-AI launch, with Prime Minister Sanae Takaichi joining by video. The Takaichi administration has made AI and semiconductors the centerpiece of a growth plan chasing ¥370 trillion ($2.3 trillion) in public and private investment by 2040. Noetra’s factory — which Nvidia bills as “the world’s first national AI infrastructure” — is the clearest bet yet. Japan’s push for independence, at least for now, rests on American chips.

In two days, Huang sat across from nearly every name that matters in Japanese tech — the CEOs of Toyota, Fanuc, Yaskawa, Fujitsu, and Kawasaki over lunch, and dozens of supply-chain chiefs over skewers and whisky in a Kanda izakaya.

It’s the same playbook he ran weeks earlier — a homecoming keynote in Taiwan, fried chicken, and a 50,000-GPU deal in Seoul last fall. This time, it was Tokyo’s turn, with the robots, the supply chain, and the chips underneath.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

source

Continue Reading

Tech

Netflix paid $587M for Ben Affleck’s AI filmmaking startup

In a new regulatory filing, Netflix revealed that it paid $587 million in cash for InterPositive, a startup co-founded by actor and director Ben Affleck.

The streaming company announced the acquisition in March, with a statement from Affleck saying he wanted to “protect the power of human creativity.” According to Affleck, InterPublic’s AI tools help filmmakers improve their footage in post-production, particularly when it comes to making up for “real-world production challenges such as missing shots, background replacements or incorrect lighting.”

At the time, Netflix announced that the entire InterPositive team would be joining the company, with Affleck joining as a senior advisor, but it didn’t disclose the financial terms of the deal. A subsequent report in Bloomberg suggested that the deal could be worth up to $600 million.

In its most recent earnings report, Netflix said that around 300 of its titles have already used generative AI.

source

Continue Reading