Overview

  • Posted Jobs 0
  • Viewed 2

Company Description

Open-R1: a Completely Open Reproduction Of DeepSeek-R1

Hey there! This article is an introduction to the job, not a claim that we’ve recreated R1 yet. We’re integrating in the open, so as quickly as we have evaluation numbers, we’ll share them. You can follow our progress on Hugging Face and GitHub.

True, however it appears like there’s absolutely nothing to be examined as of today. I presume the supreme objective is to train a new reasoning design and after that utilize the exact same evaluation metrics as o1 and the DeepSeek-R1.

Well, there must be at least some sanity check and validation to guarantee the design was trained correctly.

Oh yes, if you are speaking about the assessment number of deepseek’s design it’s coming soon!

As pointed out in the blog post there is no design called Open-R1 to test at all … not yet anyhow. This is a blog laying out that Hugging face will take the R1 Deepseek design, exercise how it was constructed as laid out in the paper and from what they released, and then replicate that procedure.

in truth this is practically how science works … A develops a plan, discovery or development and it is tested by B, C and D to see if it is reproduceable. Thats been the cornerstone of research study now for a couple of centuries.

This blog is not saying they have currently done so … Its a blog describing an intent to start training a model like R1 and calling it Open-R1.

Also DeepSeek-R1 was only launched recently, and even in their paper they outlined the calculate hours needed. While those are low compute hours for a SOTA model this does not imply you can train stated model in a week. I ‘d personally love to be able to train a transformer design in a week, but we might require to wait a while for that level of compute technology.

So there are no standards for a model that has not been built yet right? As laid out in the blog, and again in reply to your question.

However fear not, there is a GitHub Repo already and factors (hell I may join myself), some prelim work done, and a master plan. A good starting position.

n
@edbeeching
has actually examined the released designs currently

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 simply trained on o1 outputs, so jointly …/ s. This is what the new AI czars are stating

Hi! This article is an intro to the project, not a claim that we have actually replicated R1 yet. We will absolutely share the missing out on piece when we have them, you can expect the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s good and important to understand this incredible hype that does not have technical understanding and description. Science has to do with reproduction, and if they declare to be open, let them fullfill the open part.

Please do release the training cost.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will undoubtedly be striving to make certain this training dish can work for small language models on consumer hardware considering that not everyone has a cluster of H100s in your home:-RRB- The tool we used for the images was Excalidraw! https://excalidraw.com

anticipating it! WTF are your speaking about?

must be a joke

It’s really cool to see how the entire open source community comes together!

Ops …

5.5 M is number press reporter in the deepseekv3 tech report (just the training, not the experiment afaik), for R1 tough to approximate tbh but much less than 5.5 M imo

Historically, they have never ever launched code or datasets of their LLM training, so I would not expect this time to be various. If they would release it that would be remarkable of course!

Yes of course!

So essentially you’re asking to replace existing censorship with another flavour of censorship?

The code for the models are inside the model repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and developer of EQUATOR. My research study group will be dealing with a paper concentrated on duplicating particular parts of DeepSeek R1. Our goal is to reproduce the cold start and offer your group with a dataset that consists of COT and other strategies to support these efforts. We like to contribute our work to assist. Please let me understand if you discover this useful. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the assessment numbers? without it you can’t call it reproduction.

8 replies

True, however it seems like there’s absolutely nothing to be evaluated as of today. I presume the ultimate objective is to train a brand-new thinking model and after that utilize the very same examination metrics as o1 and the DeepSeek-R1.

That’s rather fascinating, I was asking myself why the questions the author exposed here are not being asked by others? I believe the work they have actually done is remarkable but at the exact same time I wonder why they would not put these missing out on pieces on if they are supposed to be totally open.
Why even without recreation and understanding of the innovation they could impact so much the market in this method?

4 replies

Hi! This article is an introduction to the project, not a claim that we have actually replicated R1 yet. We will totally share the missing out on piece when we have them, you can anticipate the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is good that we see more effort into this direction: more optimization and less strength.
Also question what tool did the author use for producing step diagram.

2 replies

Excalidraw I’m so pleased that effort like this currently exist, I’m gon na attempt to contribute:-RRB- 1 reply

anticipating it! So racist articel

2 replies

WTF are your talking about?

Awesome to have this open recreation began!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s truly cool to see how the entire open source neighborhood comes together!

Does anyone know the real training cost of r1? I can’t discover it in the paper or the statement post. Is the 6M expense reported by media simply the number taken from v3’s training expense?

2 replies

Ops …

Has anybody asked the DeepSeek team to release their training data and code, or a minimum of share them privately with an independent replication job like this? Have they turned down such a request?

A devoted replication depends upon using the exact same dataset and hyperparameters. Otherwise, any major inconsistencies with the published standards would be hard to pin down-whether due to training data distinctions or the replication method itself.

1 reply

Historically, they have never ever released code or datasets of their LLM training, so I would not anticipate this time to be different. If they would release it that would be amazing of course!

In the meantime we need to make finest guess price quotes and see if we can arrive ourselves.

You supply great duplication process of Deepseek thinking training. I will try something comparable to it.

This is really excellent info, can we fine tune with specific usage case when code is launched?

1 reply

Yes of course!

Please think about eliminating prejudiced, tainted or unaligned training data and make an effort to remove copyrighted works from the crawl from intake. This will make the model more usable. If you recycled anthropic curation checks, this may also help, remove obviouslybiased information will likely add a lot of value. We don’t desire another tainted, unaligned open source model, right? And no corporate would ever utilize deepseek or a model that reuses it, right?
We value your work for the benefit of humanity, we hope.
Miike C from NJ

1 reply

So essentially you’re asking to change existing censorship with another flavour of censorship?

Can’t wait! Hopefully the design will be uncensored but whatever you can do is alright! Love seeing open source building itself up. I’m not wise enough to actually help but I can contribute support lol

Hello guys, I am even just trying to discover code for DeepSeek-V2, in order to multi-head hidden attention. You do not seem to have code in Hugging Face even for that. Or am I missing something? Don’t see anything in src/transformers/models. MLA is not correctly described in their paper, so it would be crucial to have code for this.