I'm re-granting because I've been very impressed with the quality of Epoch's work. It's clearly far and away better than any competition when it comes to actually understanding what's going on with AI (and being publicly communicated), and several people I know who are part of Epoch strike me as highly competent. I'm also a fan of their new weekly newsletter. I haven't carefully checked the details of much of my work myself, but the one example where I thought I found an error (using total rather than active parameters of an MoE model), they then produced a newsletter with a compelling argument for why their way made more sense, which I was impressed by.

I see the main theory of impact of Epoch as broadly helping key decisionmakers in society (policy-makers, businesses, the AI safety community, etc) be well informed about AI: what has already happened, what is happening, what will happen. I'm a bit less confident in how well their projects do on the dissemination front. I see them go viral fairly often, though I am totally in a massive bubble. I've seen some scattered examples like Satya Nadella having Epoch graphs in a slide deck, which seem pretty promising!

Is this theory of impact important?

I think that AI is clearly a big deal, at this point, having an important impact on society, and expect this to accelerate. In addition to the big issues of misuse and misalignment, there's also just a lot of social and policy change that needs to happen to handle this well. By default, society changes slowly, and tech policy often makes little sense. High quality information is far from sufficient to fix this, but it helps!

I have some concern that this also has negative externalities by increasing the number of people who realise AI is a big deal and decide to race (in a similar sense to how, eg, scaling laws got more people realising the potential of AI). But my guess is that this isn't that big a deal. I think that "AI is a big deal" is pretty widely believed at this point, so that ship has largely sailed. Another concern is that this increases competitive pressures by making it clearer to people when they're losing the race. But on the other hand, this helps the person in first place realise it, which is high value (see eg the Cold War "missile gap"). And I expect key actors like AGI labs already have much better awareness of what's going on than most, so Epoch adds less value to them.

Overall, this isn't my area of expertise so I'm not confident, but my best guess is that Epoch is doing great work.

I'm only donating a token amount as I don't think this is my comparative advantage as a funder, but I would encourage others to donate, and want Epoch to have as much funding as they can productively use.

I think this is a no brainer to fund - unfaithful CoT is an important thing to understand (especially given the rise of o1-style models), this dataset seems like a significant improvement over state of the art for studying this phenomena (especially the focus on reproducibility), and the requested amount is small and seems reasonable. I have mentored Rob before and think he's competent and capable of executing on this well.

Conflicts of interest: I've mentored Rob a bunch in the past. This project specifically is largely supervised by Arthur, not me, but I may still be somewhat involved. My bar for spending time on a project is higher than my bar for funding it, so I'm not too concerned about bias/impaired judgement here, especially given the small size of the grant.

I'm making this grant largely on the recommendation of Marius (as he's much more involved in the evals field than me), thus only giving the minimum funding of $18.5K, but I'd be happy to see this fully funded!

The overall goal and plan here checks out to me and seem important: understanding how capable AI systems are seems very important, especially dangerous capabilities, both for forecasting risks and for coordinating agreements (like RSP-style conditional pauses). Current statistical practices seem pretty sloppy, and I can buy that there's a lot of low hanging fruit to improve it. I'm not familiar enough with the statistical theory here to say if the proposed method is the right call, but it seems fairly reasonable to me, and I'd be excited to see it explored properly!

The main concerns I see:

• A method is only useful if people actually use it - highly technical or complex methods seem less likely to get adoption. Though eval methods can be used by people outside AGI labs (eg Apollo), so there are far more chances for someone to try it. I expect the main things to help here are compelling evidence that it's useful, along with good explainers and software

• Getting data is expensive - I'm not sure how data efficient this method is compared to other things, but both designing questions and running an AI agent on them can be highly costly, and data efficiency seems super important

• To work, if I understand correctly, there must be a list of latent factors to study, and each question in the benchmark must be labelled according to which it uses? Existing benchmarks tend not to look like this, and so this will need to be addressed. My guess is that enumerating the relevant latent factors wouldn't be that hard, at least for a given dataset, and that another LLM could do a good job of labelling given access to the question and worked solution? But I'm not confident, and this is another point of failure

I have no conflicts of interest here

I just gave Piotr a small top-up for the purposes of running 2 hackathons (or events in the same spirit) in areas of AI Safety, eg making a satellite event for an Apart Hackathon.

I don't know Piotr much, and so can't be confident he'll execute well on this, but this seems pretty positive EV to me. This is very cheap, and I've met a surprising amount of University of Warsaw graduates in elite STEM circles (eg maths olympiads, Cambridge university, Jane Street, etc), so getting any of them excited about AI safety seems great to me. I think hackathons are a great way to energise and ground a group into actually doing practical things, rather than just reading or philosophising, and to help build momentum.

@Austin Thanks for clarifying!

Fwiw, without much context on the finances here, I'd be pro not having a 5% fee on smaller donors - my intuition is that it won't raise much money, and the discouragement of a fee would have an outsized effect on reducing donations. While larger donors are more likely to be rational about this kind of thing, and also to be able to judge how much value Manifund is actually adding to the ecosystem.

@Austin To clarify, do all Manifund grantees pay a 5% fee on donations via Manifund? Or just the grantees who are also fiscally sponsored by Manifund, and they pay it regardless of whether donations come via Manifund or elsewhere?

I'm fully funding this. I think that understanding, detecting and potentially mitigating chain of thought unfaithfulness is a very important problem, especially with the rise of o1 models. I think the approach taken here is reasonable. I think Arthur is fairly good at supervising projects, and that under him Jett and Ivan have a decent shot of making progress, and that enabling this to start a month earlier is clearly a good idea. I know Arthur well, and got a moderate amount of data on Jett and Ivan's ability to execute on this project in my MATS training program.

Conflicts of interest: Jett and Ivan were in my MATS training program (but no longer are, and I would not be involved in this project going forwards). I manage Arthur and he works on my interpretability team at GDM (but he is doing MATS in a purely personal capacity, and if anything I'm incentivised to reduce his amount of extra-curricular distractions :P ). Overall I don't feel particularly concerned about conflicts of interest here.

I like the proposal! Do you think you could productively use any more money?

I think collections like this add significant value to newcomers to the field, mostly by being a list of all areas worth maybe thinking about, and key links (rather than eg by providing a lot of takes on which areas are more or less important, unless the author has excellent taste). Gavin has convinced me that the previous post gets enough traffic for it be valuable to be kept up to date.

I'm not super convinced that a ton has changed since 2023, but enough has to be worth at least some updating, so I'm funding the MVP version (I expect this to have more errors than higher funding, but for these to largely be found in the comments, and even higher funding would still have errors). I'd be fine to see others funding it higher though

@NeelNanda Oh, I'd also love to hear more about the story behind "Oxford all time top alumni fundraiser", what does that actually mean, and how?

Interesting project! To go well, it seems like the main project person needs to be good at a few things:

1. Having a good network in the grants/funding space, across a fairly diverse range of funders (given your reply to Ryan)

   1. Plausibly, being the kind of person who could build a good network fast and has some existing connections, would suffice?

   2. Or maybe a bunch of these funders don't care as much about personal connections, and have open applications, and you can just collect info about those?

2. Being good at translation: understanding the context of funders in a range of fields, their language/culture/what they look for, and understanding the AI Safety orgs and being able to sell them effectively

Do you have much evidence that shows you're good at those two things? (Also feel free to push back against this model, or point out other key skills I am missing!)

Ah! You mean OpenAI API costs, I thought it was a weird crypto thing. I recommend clarifying this in the post

What is a token and why do you need to spend a thousand dollars on them per month to make the website work?

@ms Hmm, if it's not exposed to users, DM Austin on the Discord (linked in the corner) and ask him to fix it?

Did you intentionally make the max donation $500? Your own donation has already exceeded that, so I imagine you want to raise the cap

@michaeltrazzi +1, in particular, donating money to help you spend more time on this would feel noticeably more exciting than more time for better hardware/software/etc - I don't know if your time is fungible though Gaurav?

I've found being a MATS mentor a very valuable experience! I think my scholars have done kick-ass work, several of them have had very high impact roles going forwards, and I've mentored many more people than I would have done on my own in a way that I believe has significantly magnified my and their impact, and I appreciate MATS for facilitating this.

I'm not donating more, as MATS is a large funding opportunity that I don't think EACC is well placed for, but a token donation seems in the spirit.

I'm not sure how I feel about Lightcone as an impactful donation opportunity on the margin, but I have personally benefitted a fair bit from Lightcone's work and broadly consider it to be high quality, so feel like it's in the spirit of EACC to donate!

I haven't used Teamwork myself, but I think co-working spaces are valuable, and it seems like you do good work at impressively cheap rates!

I found these posts useful, and appreciate their existence! Especially Justified Practical Advice, and credibility of the CDC

@marisavogiatzi I'd encourage you to increase the max funding significantly higher if you would spend more hours on EA stuff given more money! It's max funding, after all. I also second Austin's suggestion of charging people some money for it (specialised to what the org can reasonably afford), this is a good way to ensure you're actually providing value.

This seems like a great service to provide! And I'd expect that someone interested in EA is better placed to provide useful work for an EA org's needs than a generic professional. It's hard to judge how good you are here, but having too much demand seems like a strong positive signal, and totally worth funding.

I've worked out of LEAH and appreciate its existence!

16 out of 49 participants doing high-impact jobs seems extremely impressive (though unclear how much I'd agree with your definition of "high-impact"!) - I'd love to clarify what exactly you mean by that

I think this is a clearly useful public good, and I've found the results of the previous survey useful at random points in various minor ways

I think 80K is doing good work that should clearly be funded - I've had 80K career advising several times and found it quite valuable, and think it helped push me over the edge into pursuing AI Safety work (not donating more as I think they're far too large an org to get much value from EACC, and it's best spent on smaller orgs)

I think GWWC is doing good work, and I value there being an org carrying the torch of effective giving (not more as I think they're far too large an org to get much value from EACC, and it's best spent on smaller orgs)

Note: I currently see this on the Manifund main page:

Title: EACH / CFI Community

Summary: Growth Fund

I think it'd be good to add more detail, eg expanding "EA for Christians", since you may miss interested donors who are skimming the 60+ opportunities!

The focus on animal charities is a key detail and easy to miss, I'd recommend putting it in the title and project summary

Note: Your max funding here is $500, which I presume is an error? (If you can't fix it yourself, I imagine you can DM Austin on Discord to fix it)

@NeelNanda I was also impressed that Alex was able to defend the case for the project in quite a lot of detail, had already thought of several experiments I suggested, and generally seemed to care a lot about baselines and rigour.

I'm also generally pro supporting the work of promising junior researchers regardless of the project to help them build skills and credibility.

I discussed this with Alex Cloud. I'm somewhat pessimistic about whether the technique will both work and not have a crippling alignment tax, but he made a pretty compelling case that it MIGHT, and could be a big deal if it worked, and it's a fairly elegant idea that seems like it has potential for some cool things even if the exact proposal doesn't work.

Either way, this was a fairly cheap grant, a small fraction of the cost of labor going into the project, and it seems valuable to gather more data on whether the technique works and I expect that having more compute will make the quantity and quality of the evidence better, especially if they can go beyond using tiny stories to more realistic settings. There were several experiments Alex and I agreed would be good ideas, and I would be keen to see them happen.

I think Decode do great work, and I suggested they submit this here. I expect to fund at least part of it, and am chatting details with them.

How long a time period would the 45K requested cover? I'm very surprised you can pay 0.5 FTE and rent a decently sized office space for so little, what's the breakdown here?

And what kind of people (eg what roles/working for which orgs) work out of the space at the moment, or in the expected future?

What evidence do you have from the first iteration of the program on how well it went? (In particular, assessment of how much counterfactual value the program added - participants who go on to eg do MATS may have gotten into it anyway). Eg did you survey participants after the program/several months later? (I know it was only a few months ago, so you don't get too much data).

I looked through your application, and the website, but still don't have a great sense of this

I had previously discussed this grant with Lovis and suggested he apply.

Why is this a good idea?

I think Sparse Autoencoders are one of the most promising areas of mech interp work right now. Better understanding SAE circuits seems exciting, and I think that understanding the circuit required to produce a feature is an important direction. This is both a sub-part of the broader project of finding end-to-end circuits, and could help with interpreting what a feature does (especially important features like the safety relevant features in Scaling Monosemanticity) - I would be very excited if this project finds case studies of features that have ambiguous maximum activating examples, but the meaning is clarified by studying a circuit.

(Note that the applicants shared me on a more detailed project proposal than what was shared publicly, which I broadly think was sensible, though I disagreed on some points)

Concerns

• Research is hard, and there's a good chance this project doesn't really go anywhere interesting

• This is a hard and somewhat open-ended question, though I think they had some decent ideas of concrete entry points

• There's many directions the project could go in, and it'd be easy to get caught in rabbit holes/constantly flit between things and never do any of them properly.

Why this amount?

This was the salary requested, I think somewhat pegged to academic summer researcher salaries, which are a fair bit lower than the market rate for independent researchers, so no complaints from me. The compute may not be needed, since the lab provides some, but it would be silly for the project to be bottlenecked by lacking compute. This overall seems like a fairly small grant, with some chance of going somewhere interesting, and so a pretty obvious accept.

Conflicts of interest

Lovis is one of my MATS alumni, but we haven't been working together for several months, so I don't feel too concerned about the conflict of interest, and it means I have a fair amount of data to evaluate him. I don't personally benefit from this project (except in that all good mech interp research helps my own work!), and don't anticipate being a co-author on any papers produced

Is this being run by any particular group or organisation?

Advocacy, R&D, and field-building seem like very different things for such a small and new org to be trying to do at once. Why did you make this decision, and how concerned are you about being spread too thin?

You also might want to add to his bio that Alexandre was second author on the Indirect Object Identification paper, which I think was great work.

@AdamGleave Just noting that I was quite impressed by the paper that came out of this ( https://arxiv.org/abs/2403.19647 ) - good grant, and good work by Sam, Can and co!

Main points in favor of this grant

I think that SAEs are a big deal in interpretability, with lots of valuable interp work that can be unlocked with good SAEs. Developing, understanding and using SAEs is the major focus of both Anthropic's mech interp team and my team (Google DeepMind mech interp). I feel like SAE training is currently very janky and pre-paradigmatic and I would love to see progress here.

Why grant to Glen? I was particularly impressed by the ProLU work. Though it was, unfortunately, highly similar to my team's Gated SAE work, making the actual impact lower, I think ProLU was a good and principled idea that correctly identified a flaw in SAE training, and empirically showed that it was a significant improvement. Further, I think Glen broadly did the right things to show that it was an improvement, and did the leg work of training a bunch of SAEs on a range of models, layers and sites (though was bottlenecked on compute I think) and carefully comparing Pareto frontiers - this makes me more optimistic that if Glen finds an important improvement, he'll present enough evidence for me to believe him! I thought the write-up was pretty rough, but it was quite rushed, so that's not a major consideration.

We had a call, and I thought Glen was thinking about things sensibly. In particular, he had a strong emphasis on iterating fast, building the infra to try out many ideas quickly, and doubling down on any idea that meets a moderately high quality bar. I think this is a great way to do this kind of research. Another good sign is that Glen said ProLU felt less interesting to him than some of his other ideas, but had better empirical results, so was higher priority and he doubled down on it - being willing to be pragmatic like this and prioritise results makes this kind of research go much better!

Donor's main reservations

Even with a grant, this kind of research is much easier to do inside a lab, where you have a lot of compute, and more engineering expertise. There are people in labs working on this, eg Anthropic has a several person sub-team on science + scaling of SAEs. But there's many problems to work on, and ultimately not many researchers working on it, and Glen seems to have many interesting ideas, so I'm not too concerned about this. There is risk of duplicate work, eg ProLU and Gated SAEs, but I don't think that's a strong enough consideration to sink the grant.

I'm generally pretty wary of people doing independent research, especially junior researchers, with concerns specifically around lacking structure, accountability, motivation, feedback/mentorship, and stability. Glen says he hasn't been experiencing any issues with executive function, which is great! I've encouraged him to look for collaborators, and ideally a mentor, which would make me feel much better about the grant. It doesn't sound like independent research is his long-term plan, which makes me feel better about this.

Glen doesn't have much of a research track record, making it hard to be confident in this going well. But he seems promising, and I think it's good to give promising, inexperienced researchers a chance to prove themselves.

I have some concerns that this grant could result in a bunch of half-baked research threads, with no public write-up or clear conclusions. But Glen seems pretty motivated to make that not happen, and I think he also has a strong incentive to produce something legible and cool to eg help with future grant/job apps

Process for deciding amount

I'm honestly pretty confused about how to think about grant amounts here. $9K/month seems not crazy salary for someone living in SF, but I'd happily follow default rates for independent researchers if anyone has compiled them! $2K/month for compute seems enough to make it not a bottleneck without being too big a fraction of the grant. I'm funding this up to 5 months to balance between wanting Glen to have runway and a chance to prove himself, and wanting to see results before I recommend a larger/longer grant. If other grantmakers are excited about Glen's work I'd be happy to see them donating more though.

Conflicts of interest

Glen did my MATS training program about 6 months ago. I do a lot of SAE research, and expect to benefit from better knowledge of SAE training, but in the same way that the whole community will!

@NeelNanda Note: Tom and I discussed this grant before he applied here, and I encouraged him to apply to Manifund since I thought it was a solid grant to fund.

@Austin Yep, I'd be happy to pay salary on this if Tom wants it (not sure what appropriate rates are though). Tom and I discussed it briefly before he applied.

Main points in favor of this grant

I think that determining the best training setup for SAEs seems like a highly valuable thing to do. Lots of new ideas are arising about how to train these things well (eg Gated SAEs, Prolu, Anthropic's April update), with wildly varying amounts of rigour behind them, and often little effort put into replicating them and seeing how they combine. Having a rigorous and careful effort doing this seems of significant value to the mech interp community.

Tom is a strong researcher, though hasn't worked on SAEs before, I thought the Hydra Effect and Understanding AlphaZero were solid papers. Joseph is also solid and has a lot of experience with SAEs. I expect them to be a good team.

Donor's main reservations

The Google DeepMind mech interp team has been looking somewhat into how to combine the Anthropic April Update methods and Gated SAEs, and also hopes to open source SAEs at some point, which creates some concerns for duplicated work. As a result, I'm less excited about significant investment into open source SAEs, though having some out (especially soon!) would be nice.

This is an engineering heavy project, and I don't know too much about Tom's engineering skills, though I don't have any reason to think they're bad.

Process for deciding amount

As above, I'm less excited about significant investment into open source SAEs, which is the main reason I haven't funded the full amount. $4K is a fairly small grant, so I haven't thought too hard about exactly how much compute this should reasonably take. If the training methods exploration turns out to take much more compute than expected, I'd be happy to increase it.

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

Tom and I somewhat overlapped at DeepMind, but never directly worked together.

Joseph is one of my MATS alumni, and currently doing my MATS extension program. I consider this more of a conflict of interest, but my understanding is that Tom is predominantly driving this project, with Joseph helping out where he can.

I expect my MATS scholars to benefit from good open source SAEs existing and for both my scholars and the GDM team to benefit from better knowledge on training SAEs, but in the same way that the whole mech interp ecosystem benefits.

"resulting in three publications accepted at top-tier academic ML venues (NeurIPS, ACL, ICLR),"

To add context in case people get misled by this line, the NeurIPS and ICLR papers (N2G here) were workshop papers, as far as I can tell, not main conference papers. For people not in ML, a conference like NeurIPS or ICLR has both conference papers (one of the highest status ways to publish in ML) and workshop papers (lower prestige and less selective, I'd roughly say a workshop paper is 1/3-1/2 as impressive as a conference paper).

To me, the prior is that most hackathon projects are a total flop and don't go anywhere, so helping someone convert it to a workshop paper is still impressive! (But main conference would have been very impressive). And the ACL paper was a main conference paper, which is impressive!

This seems pretty worth funding to me - it's a cheap grant, and I think this would be a cool paper to exist! I don't have a background in neuroscience or cognitive science, and I expect there's some techniques there worth my knowing about that would be useful for my work, but that much of it is irrelevant. I'd love for a paper surveying and summarising the most relevant ideas to exist! I've mentored Wes Gurnee and I trust his judgement/ability to represent the mech interp side, and expect Stephen Casper to also give good takes here. I don't know the rest of the organisers, but Wes vouches for their overall competence. I'd fund this myself if I had a regranting budget.

(I think a Nature publication is very ambitious, and would advise against bothering, but think an Arxiv publication is more than sufficient to make this worthwhile)

Lawrence is great, very experienced with alignment, and I trust his judgement, this seems like a great thing to fund! I would donate myself if this was tax deductible in the UK (which I don't think it is?)