My big mining rig project
-
[quote name=“Nutnut” post=“2490” timestamp=“1368276831”]
LOL - my brother is adamant that i’ll get raided by the cops coz the chopper will think i’m growing weed in there!
[/quote]Was only thinking about this yesterday ;D
Some great ideas in these posts that I can use. My thanks to all.
-
Note to self: Use your own damn start scripts, you made them for a reason!
I was plagued with tons of HW errors. Turns out I simply didn’t run these lines after my reboots:
export DISPLAY=:0
export GPU_USE_SYNC_OBJECTS=1
export GPU_MAX_ALLOC_PERCENT=100*double facepalm*
-
So, some 3D Models here for your viewing pleasure.
[img]http://imageshack.us/a/img716/9792/miningrig01.jpg[/img]
A first general view. The blocks on the right are the power supplies, the one on the left the GPUS. The motherboard is behind the GPUs and those strange things at the bottom are two 230mm fans. There are these two metal sheets that angle towards the GPUs, that’s some ducting for airflow.
[img]http://imageshack.us/a/img560/5007/miningrig02.jpg[/img]
From the other side. The PSU on the left sucks in air through the bottom (large hole in the upper one) and pushes it out towards the back of the PSU (the lower one is in a standard position, the upper blows the hot air upwards). Also missing here is the back plate that holds all of this together. The mainboard is floating, as is the upper PSU. There is a plate that will cover the space from where the gpus are towards the back. I just removed it for displaying purposes.
[img]http://imageshack.us/a/img9/2857/miningrig03.jpg[/img]
Last but not least a close up of the ducting. The air gets pushed towards the cards. On each side (now open) there is another piece that stops air from going sideways, again removed to see everything easier. This setup also shields the mainboard from some extra heat. I was thinking about adding a side fan that just blows over the mainboard for a bit extra cooling of that area and to make sure not too much heat form the card gets to the mainboard.
Overall the rig is now 42 cm tall, 70 wide and 28 cm deep.
Not perfectly happy with it though. The cards are very close imho and that can lead to troubles. The cables will reach, they are at the most 16 cm from connector to plug, but it’s really stretching the outer limits of 19 cm that the cables can do. I still don’t have them in my hands, so that makes it a bit harder to fully plan this out.
I will store this version and start drawing up an alternative. Maybe I can come up with something even better.
-
Would placing a PSU on each side help?
Also to space the cards out more, you could use 2 pci riser cables for per card for cards furtherest away from the motherboard. -
Yes, placing one on each side was one of the modifications I had in mind. The cables seem to be rather short otherwise. Stacking risers is something I try to avoid, though I might not really need to, since we are talking about PCIE x1 here and the data that goes through there is rather low.
On another note: Powered risers seem to be a hand made thing, nobody sells them outright except cablesaurus and they don’t seem to do that anymore either. Looking at their picture, you can see very clear that it’s soldered on as an afterthought with only giving it some extra 12V juice. I have to say that sounds a bit odd to me, since GPUs draw the majority of their 12V power already from the external connectors. I maybe will run it on my other motherboard that has some nice tabs to read the voltage on the pcie bus easy. But that is for another time, once I have risers and maybe a powered one too. Either way, if you can’t buy one, it seems rather trivial to make one.
-
The journey continues.
To make everything a bit more stable I put the whole thing int an ATX case for now and added a second GPU. With two 7950’s running at 560 Kh/s each the system draws 580 - 590 Watts. Adding a third card would probably still be possible though pushing the PSU. Card 4 could mean disaster.
So I ordered some risers and cables to make powered risers out of them. I already have simple 1x risers on the way, which I could abuse (and destroy) for this task, since the x1 to x16 risers are just unconnected at the back part anyways.
A curious discovery on heat has happened though. Directly connected to the board, there is one free slot between these dual slot cards. Now the card whose fans are sucking in air from the other cards back seem to emit heat out back to the card below as well. Something I noticed in the single card setup, where there is heat leaking out around the fan, not just down and up. My design idea is now to create a heat barrier between cards.
-
Since I have the cards now in my hands and a better understanding and feel for them, I decided to redo the model for them in 3D. I also decided to leave off the bracket as it just hinders airflow with my new plans. Here a peek at how the model looks like:
[img]http://img198.imageshack.us/img198/3909/club3damdradeonhd7950no.jpg[/img]
Now for the next part, the heat shield.
-
[quote name=“ChristianRiesen” post=“2974” timestamp=“1368433750”]
My design idea is now to create a heat barrier between cards.
[/quote]
I have been thinking about that also, would ceramic fibre be a possible material to use as a heat shield, or would that be overkill? -
Really awesome man. Good luck with your project. Please post some pics once it’s up.
-
Pretty cool! But do you think it will ever pay for itself at this point? :-\
-
[quote name=“atariguy” post=“3075” timestamp=“1368474544”]
Pretty cool! But do you think it will ever pay for itself at this point? :-\
[/quote]
This is mainly an engineering challenge for me, something for fun, but also trying out things which might be useful to others. If I got a good design, I could create multiple of these nodes and I would put the plans for it public so anyone could build them.So far the actual money I invested is very little. I put a lot of time into it to get it right and then I will put more time into it to make it better :)
-
[quote name=“pyxis” post=“3069” timestamp=“1368472326”]
[quote author=ChristianRiesen link=topic=387.msg2974#msg2974 date=1368433750]
My design idea is now to create a heat barrier between cards.
[/quote]
I have been thinking about that also, would ceramic fibre be a possible material to use as a heat shield, or would that be overkill?
[/quote]
I was more thinking about metal sheets. They themselves will act as heatsink in a fashion. I don’t want the casing to deflect or worse, reflect heat, I want it to make sure it doesn’t get blown hard on neighboring components. That way, and with a good fan, I probably could reduce the heat significantly. That in turn means longer lifetime of the components. -
So made a second design.
[img]http://imageshack.us/a/img17/3933/rig4p.jpg[/img]
[img]http://imageshack.us/a/img826/551/rig3.jpg[/img]This shows 8 cards but I might just make it with 6 or 7. These round holes are representations for 120 mm fans. Each one is high volume high velocity and is then fed into a construct like this:
[img]http://imageshack.us/a/img41/5461/gpucasing2.jpg[/img]
That (relatively) evenly splits the fans power into 4 and feeds that through the channel for each card, to blow it out the back. You can also see there is no bracket on the cards, to make sure air gets easier through.
I thought about removing the plastic cowls from the cards, but they are attached by 4 screws I could only remove if I removed the entire cooler. Not gonna do that. I could get it back on properly, but I’d rather not. Also don’t want to loose the fan control option of the cards, some might get hotter than others.
[img]http://imageshack.us/a/img40/1263/rig1t.jpg[/img]
And lastly here a view into the bottom part. I removed the side panel so it’s easier to see, also didn’t include a few extra struts to make it more stable on the inside. The tabs hanging down (the lowest hanging ones) are representing the cable starting there from the risers. Inside the tunnels is actually just the PCIE plug.
The overall strategy is to keep heat away from the mainboard and the other cards, get it pushed out the back as efficiently as possible and then also make the entire thing stack-able. That last one was impossible (of course) with my first design. I will mock up something smaller with metal and try it out as soon as I have the correct risers present. I’m very interested to see how it does temperature wise.
-
That is much better looking, it will be interesting to see what the temperature readings are with and without heatsinks added.
-
The power supplies still bother me to be honest, for multiple reasons.They are constructed to suck in air at the bottom and expel through the back. I put them on their sides like that so they will push the hot air out the back like the rest of the rig. Trouble is, it will suck in air very closely from where it pushes it out. Next comes the problem with switches. If one is switched off, the entire rig will fail. I was thinking of opening them and removing switches, none of which appeals. The fans blowing over the cards also have their own switch (they are directly AC, in order to not push on the already taxed power supplies).
I will try one more alternative.
-
[img]http://img19.imageshack.us/img19/3218/rigv3variants.jpg[/img]
Leaving off all kinds of covers just for displaying purposes. At that back there would be a cover ov course where only the holes from the PSUs come through below the GPUS. It would have some covers on the sides and some sort of pedestal for the mainboard (the HDD could be mounted below it for easy access). In both designs, the PSU fans would pull the hot air from the insides and expel them out the back. The intake of the air would be way on the other side, where the case is open and not where the exhaust of the GPUs is (or worse, of a second rig below it).
I toyed with the idea of flipping the entire rig to one side or the other. Neither option looks very appealing to me. If the cards face down (fans) then they do the same drooping they do in current systems. Look at your computer, you will see how the card is slightly flexed. The same is true for the other direction. I do intend to add another fastener at the back of the card (towards the 120mm fans) but not yet sure how exactly.
Maybe one or two more iterations.
From a manufacturing point of view, I found a shop that can lasercut the pieces I need, all for a rather reasonable price. So in the end, I just have to stick it together a bit like Lego and hit it with some welding points.
-
While staring at the previous pictures something hit me. By normal standards, power is needed at the bottom near the main board. I was so in trance with that, until I started making the holes needed for the PCIE plugs, at the top of the case. Now suddenly the game changed. The power is needed on top, not at the bottom. The bottom only needs mainboard supply and the harddisk. So here comes the latest iteration:
[img]http://imageshack.us/a/img405/8861/rigv43.jpg[/img]
[img]http://imageshack.us/a/img402/3631/rigv44.jpg[/img]Both power supplies sit comfortably at the top. They have each a spacer below them so they have more stability and can suck in fresh air easier. Getting power to the cards is very fast and simple, it’s right there in front of each supply. The ATX connector needs to go down though and that it can in between the two right most fans. These are high flow 12v fans at 140mm. Coincidentally, the box the gpus sit in is 150mm tall on the inside, so the fans blow pretty much through the entire spaces for all 7 cards.
The gpu casing has space for 7 gpus and is made to have the cards sit in the risers comfortably, getting the power through a hole from above and be mounted with two screws to the side. The gpus brackets are gone and this allows for easier airflow out the back.
[img]http://imageshack.us/a/img515/9805/rigv45.jpg[/img]
The fans are sitting on this holder, which can be installed no matter the orientation, so no problem if its upside down. It’s held with two thumbscrews on top and two at the bottom in place. That way the GPU holder is still easy accessible for maintenance and installing.
The mainboard sits on a plate that also has the harddisk screwed to it. I’ll let the laser place cut holes into it for 3.5 and 2.5 disks, in case I change my mind and switch to something else later. The plate is screwed on to the frame with 3 thumbscrews.
That frame is made of 15 mm square hollow tubing. Each corner is open to the top and to the bottom. That way a second unit could be put on top of it (with a small spacer in between to make sure it wont slip of sideways).
The entire thing is now 497 mm wide, 360 mm deep and 374 mm tall. If you stack 5 of these on top of one another, with a generous spacer of 2 mm, the get a tower 188 cm tall.
One such unit runs on an approximated 1800 W and produces 4550 Kh/s for Feathercoin.
Now I need the risers and fans and I will try out how well those fans can actually cool the cards and how much space I need to cut out for the risers. Then it’s all off to the laser cutting shop.
-
Now that is thinking outside the box ;)
-
Yeah, sort of :)
I was thinking about adding panels, but what for? It’s not about esthetics, even though I did some small adjustments to make it look a bit nicer here and there, which did never hinder functionality or might even make it better over all.
The most important part is the gpu box in the center which will be welded to the bar at the back that it rests on and to the bar on each side along it. The motherboard and harddisk are on their own “tray” so it can be removed for maintenance, which also makes access to the underside of the gpu box easier. The psu cover is there so the units can be mounted (and a bit as a heat shield form the heat below being pushed out). But other than that, the PSUs are pretty much open air. And lastly the fan mount which can be removed for easy access to the gpus.
You might note that the fans are actually attached to the outside of the fan mount, not on the inside “tucked away”. This has multiple reasons. If they where on the inside the space around them (small they may be) can create some undesirable airflow. It’s not huge but was a starting point. The next is a heat concern. The box and the fan mount both work as an extended heat sink in a way. The fan encased in this construct would be subjected to more heat than strictly needed, which lowers the lifetime of the fan. and lastly an organization part. The space between the middle and the outer fans, allows for cables to be run from the top to the bottom while keeping them out of the way of, well, everything. Pretty much only needs 3 cables going that way anyways, 24 pin ATX, the 8 pin plug for the mainboard and one power plug for the hard drive.
I might make the fan mount a bit larger, mainly longer, to bring the fans further away from the box, but that I need to first test with the fans once they arrive.
Also still needs finetuning for the motherboard tray (position of disk, mainboard holes…) and the mounting of the risers on the box and the card inside the box is not yet 100% done either. Those should be small things to do though and once done, I can finish the layout of the parts and send them off to be cut.
-
Where are you putting the rig? If its in an outbuilding/shed have you considered making an exhaust manifold from the fans with large bore ducting to the outside world? That lot looks like it’ll kick out a serious amount if heat!