AI Top-of-Mind for 9.17.24 — Chip Wars

Today: ByteDance and SMIC, deeper dives into OpenAi’s o1, some good tutorials, and barriers to marketing AI adoption

Top-of-mind is another deep dive into AI hardware, looking into both ByteDance and SMIC. First up, ‘The Information’ describes the firm’s efforts to develop its own AI chips in conjunction with TSMC, avoiding US sanctions and providing a lower-cost alternative to Nvidia. Both the training and inferences chips are planned to be ready by 2026. On competition:

· China’s largest tech giants, including Tencent, Alibaba and Baidu, have slashed prices for using their models by as much as 97%.

· This year, ByteDance announced a batch of low-cost LLMs, some of which it priced as much as 99% less than comparable offerings from OpenAI for generating a token

And on compliance with US export restrictions, note that the sanctions only apply to single chips, and there are creative ways to interconnect multiple chips into clusters, boosting performance.

Within China, SMIC (Semiconductor Manufacturing International Corporation), the domestic competitor to TSMC, mentioned above, is the 800-pound gorilla for chip manufacturing, shipping over four million wafers in just the first half of 2024. The ‘NY Times’ offers a glimpse into how the company that provides chips to Huawei and Qualcomm operates, as well as its strategy. And although SMIC doesn’t have access to the latest technology from ASML and others, as noted above, a cluster of lower-performance chips can be just as effective, especially if delivered at a massive discount to Nvidia.

“The fence has gotten higher, but we’ve decided to leave open the front, side and back gate,” said Jimmy Goodrich, a senior adviser for technology analysis to the RAND Corporation.

Source: NY Times

Moving to models, ‘The Information’ takes another look at OpenAI’s o1, both the positive and the negative. From the reporting:

· Among the highest praise came from Terence Tao, a preeminent mathematician and professor at UCLA. He said o1-preview was like “trying to advise a mediocre, but not completely incompetent, graduate student. However, this was an improvement over previous models, whose capability was closer to an actually incompetent graduate student.”

· He could see future models acting like a competent grad student, “at which point I could see this tool being of significant use in research level tasks.”

· In other ways, o1-preview falls short. One early tester told me that it struggles with long questions, meaning that the questions have to be broken down into several parts. OpenAI itself has admitted that o1-preview is on par with, or even worse than, GPT-4o in some cases, such as writing or editing text. And o1-preview still gets stumped by some simple puzzles that any middle schooler could solve.

· All this suggests that the o1-preview release may have been rushed, either because of the company’s ongoing fundraising efforts or because of growing pressure from competitors. We should also point out that there’s a fuller, better version of o1-preview (it’s just called o1) that OpenAI didn’t launch but still published evaluation results about it.

Continuing with o1, Datadrifters looks at the architecture of the model as well as demonstrated performance. Some observations from the post:

People have speculated that o1 achieves this through advanced techniques such as Chain-of-Thought (CoT) or Tree-of-Thoughts (ToT) frameworks, where the model internally reasons step-by-step.

There is also talk of the model employing an “inner monologue”, using special tokens or tags to indicate when deeper reasoning is required.

This could involve the model emitting a <thinking> tag to trigger additional processing or engage different sub-models.

If you already played with it, you can see several key components:

· Using Templated Chains-of-Thought: Guiding reasoning through structured prompts.

· Generalizing Policies: Handling a variety of problems with adaptable strategies.

· Ground-Truthing Processes: Improving reasoning rather than relying solely on memorized knowledge.

And now turning to tutorials, ‘CB Insights’ is offering a recorded webinar on AI agents, with these topics:

• The limitations of AI agents
• How companies are tackling reliability challenges
• The applications gaining traction
• Where we’re seeing enterprise adoption of agents

And another from Edward Bullen that does a good job of explaining the foundations including RAG, agents, open source, vector databases, and more. He includes a very telling graphic:

And, just out, another ‘Dummies’ book, this time from ‘Snowflake’ on Generative AI and LLMs. Chapters include:

· Introducing Gen AI and the Role of Data

· Understanding Large Language Models

· Understanding Large Language Models

· Bringing LLM Apps into Production

· Reviewing Security and Ethical Considerations

· Five Steps to Generative AI

And lastly, on the marketing front, are we doing all we can to train marketeers to properly leverage AI tools? ‘MarketingProfs’ published the results of a recent survey that says otherwise. From the article: