My big mining rig project
-
The journey continues.
To make everything a bit more stable I put the whole thing int an ATX case for now and added a second GPU. With two 7950’s running at 560 Kh/s each the system draws 580 - 590 Watts. Adding a third card would probably still be possible though pushing the PSU. Card 4 could mean disaster.
So I ordered some risers and cables to make powered risers out of them. I already have simple 1x risers on the way, which I could abuse (and destroy) for this task, since the x1 to x16 risers are just unconnected at the back part anyways.
A curious discovery on heat has happened though. Directly connected to the board, there is one free slot between these dual slot cards. Now the card whose fans are sucking in air from the other cards back seem to emit heat out back to the card below as well. Something I noticed in the single card setup, where there is heat leaking out around the fan, not just down and up. My design idea is now to create a heat barrier between cards.
-
Since I have the cards now in my hands and a better understanding and feel for them, I decided to redo the model for them in 3D. I also decided to leave off the bracket as it just hinders airflow with my new plans. Here a peek at how the model looks like:
[img]http://img198.imageshack.us/img198/3909/club3damdradeonhd7950no.jpg[/img]
Now for the next part, the heat shield.
-
[quote name=“ChristianRiesen” post=“2974” timestamp=“1368433750”]
My design idea is now to create a heat barrier between cards.
[/quote]
I have been thinking about that also, would ceramic fibre be a possible material to use as a heat shield, or would that be overkill? -
Really awesome man. Good luck with your project. Please post some pics once it’s up.
-
Pretty cool! But do you think it will ever pay for itself at this point? :-\
-
[quote name=“atariguy” post=“3075” timestamp=“1368474544”]
Pretty cool! But do you think it will ever pay for itself at this point? :-\
[/quote]
This is mainly an engineering challenge for me, something for fun, but also trying out things which might be useful to others. If I got a good design, I could create multiple of these nodes and I would put the plans for it public so anyone could build them.So far the actual money I invested is very little. I put a lot of time into it to get it right and then I will put more time into it to make it better :)
-
[quote name=“pyxis” post=“3069” timestamp=“1368472326”]
[quote author=ChristianRiesen link=topic=387.msg2974#msg2974 date=1368433750]
My design idea is now to create a heat barrier between cards.
[/quote]
I have been thinking about that also, would ceramic fibre be a possible material to use as a heat shield, or would that be overkill?
[/quote]
I was more thinking about metal sheets. They themselves will act as heatsink in a fashion. I don’t want the casing to deflect or worse, reflect heat, I want it to make sure it doesn’t get blown hard on neighboring components. That way, and with a good fan, I probably could reduce the heat significantly. That in turn means longer lifetime of the components. -
So made a second design.
[img]http://imageshack.us/a/img17/3933/rig4p.jpg[/img]
[img]http://imageshack.us/a/img826/551/rig3.jpg[/img]This shows 8 cards but I might just make it with 6 or 7. These round holes are representations for 120 mm fans. Each one is high volume high velocity and is then fed into a construct like this:
[img]http://imageshack.us/a/img41/5461/gpucasing2.jpg[/img]
That (relatively) evenly splits the fans power into 4 and feeds that through the channel for each card, to blow it out the back. You can also see there is no bracket on the cards, to make sure air gets easier through.
I thought about removing the plastic cowls from the cards, but they are attached by 4 screws I could only remove if I removed the entire cooler. Not gonna do that. I could get it back on properly, but I’d rather not. Also don’t want to loose the fan control option of the cards, some might get hotter than others.
[img]http://imageshack.us/a/img40/1263/rig1t.jpg[/img]
And lastly here a view into the bottom part. I removed the side panel so it’s easier to see, also didn’t include a few extra struts to make it more stable on the inside. The tabs hanging down (the lowest hanging ones) are representing the cable starting there from the risers. Inside the tunnels is actually just the PCIE plug.
The overall strategy is to keep heat away from the mainboard and the other cards, get it pushed out the back as efficiently as possible and then also make the entire thing stack-able. That last one was impossible (of course) with my first design. I will mock up something smaller with metal and try it out as soon as I have the correct risers present. I’m very interested to see how it does temperature wise.
-
That is much better looking, it will be interesting to see what the temperature readings are with and without heatsinks added.
-
The power supplies still bother me to be honest, for multiple reasons.They are constructed to suck in air at the bottom and expel through the back. I put them on their sides like that so they will push the hot air out the back like the rest of the rig. Trouble is, it will suck in air very closely from where it pushes it out. Next comes the problem with switches. If one is switched off, the entire rig will fail. I was thinking of opening them and removing switches, none of which appeals. The fans blowing over the cards also have their own switch (they are directly AC, in order to not push on the already taxed power supplies).
I will try one more alternative.
-
[img]http://img19.imageshack.us/img19/3218/rigv3variants.jpg[/img]
Leaving off all kinds of covers just for displaying purposes. At that back there would be a cover ov course where only the holes from the PSUs come through below the GPUS. It would have some covers on the sides and some sort of pedestal for the mainboard (the HDD could be mounted below it for easy access). In both designs, the PSU fans would pull the hot air from the insides and expel them out the back. The intake of the air would be way on the other side, where the case is open and not where the exhaust of the GPUs is (or worse, of a second rig below it).
I toyed with the idea of flipping the entire rig to one side or the other. Neither option looks very appealing to me. If the cards face down (fans) then they do the same drooping they do in current systems. Look at your computer, you will see how the card is slightly flexed. The same is true for the other direction. I do intend to add another fastener at the back of the card (towards the 120mm fans) but not yet sure how exactly.
Maybe one or two more iterations.
From a manufacturing point of view, I found a shop that can lasercut the pieces I need, all for a rather reasonable price. So in the end, I just have to stick it together a bit like Lego and hit it with some welding points.
-
While staring at the previous pictures something hit me. By normal standards, power is needed at the bottom near the main board. I was so in trance with that, until I started making the holes needed for the PCIE plugs, at the top of the case. Now suddenly the game changed. The power is needed on top, not at the bottom. The bottom only needs mainboard supply and the harddisk. So here comes the latest iteration:
[img]http://imageshack.us/a/img405/8861/rigv43.jpg[/img]
[img]http://imageshack.us/a/img402/3631/rigv44.jpg[/img]Both power supplies sit comfortably at the top. They have each a spacer below them so they have more stability and can suck in fresh air easier. Getting power to the cards is very fast and simple, it’s right there in front of each supply. The ATX connector needs to go down though and that it can in between the two right most fans. These are high flow 12v fans at 140mm. Coincidentally, the box the gpus sit in is 150mm tall on the inside, so the fans blow pretty much through the entire spaces for all 7 cards.
The gpu casing has space for 7 gpus and is made to have the cards sit in the risers comfortably, getting the power through a hole from above and be mounted with two screws to the side. The gpus brackets are gone and this allows for easier airflow out the back.
[img]http://imageshack.us/a/img515/9805/rigv45.jpg[/img]
The fans are sitting on this holder, which can be installed no matter the orientation, so no problem if its upside down. It’s held with two thumbscrews on top and two at the bottom in place. That way the GPU holder is still easy accessible for maintenance and installing.
The mainboard sits on a plate that also has the harddisk screwed to it. I’ll let the laser place cut holes into it for 3.5 and 2.5 disks, in case I change my mind and switch to something else later. The plate is screwed on to the frame with 3 thumbscrews.
That frame is made of 15 mm square hollow tubing. Each corner is open to the top and to the bottom. That way a second unit could be put on top of it (with a small spacer in between to make sure it wont slip of sideways).
The entire thing is now 497 mm wide, 360 mm deep and 374 mm tall. If you stack 5 of these on top of one another, with a generous spacer of 2 mm, the get a tower 188 cm tall.
One such unit runs on an approximated 1800 W and produces 4550 Kh/s for Feathercoin.
Now I need the risers and fans and I will try out how well those fans can actually cool the cards and how much space I need to cut out for the risers. Then it’s all off to the laser cutting shop.
-
Now that is thinking outside the box ;)
-
Yeah, sort of :)
I was thinking about adding panels, but what for? It’s not about esthetics, even though I did some small adjustments to make it look a bit nicer here and there, which did never hinder functionality or might even make it better over all.
The most important part is the gpu box in the center which will be welded to the bar at the back that it rests on and to the bar on each side along it. The motherboard and harddisk are on their own “tray” so it can be removed for maintenance, which also makes access to the underside of the gpu box easier. The psu cover is there so the units can be mounted (and a bit as a heat shield form the heat below being pushed out). But other than that, the PSUs are pretty much open air. And lastly the fan mount which can be removed for easy access to the gpus.
You might note that the fans are actually attached to the outside of the fan mount, not on the inside “tucked away”. This has multiple reasons. If they where on the inside the space around them (small they may be) can create some undesirable airflow. It’s not huge but was a starting point. The next is a heat concern. The box and the fan mount both work as an extended heat sink in a way. The fan encased in this construct would be subjected to more heat than strictly needed, which lowers the lifetime of the fan. and lastly an organization part. The space between the middle and the outer fans, allows for cables to be run from the top to the bottom while keeping them out of the way of, well, everything. Pretty much only needs 3 cables going that way anyways, 24 pin ATX, the 8 pin plug for the mainboard and one power plug for the hard drive.
I might make the fan mount a bit larger, mainly longer, to bring the fans further away from the box, but that I need to first test with the fans once they arrive.
Also still needs finetuning for the motherboard tray (position of disk, mainboard holes…) and the mounting of the risers on the box and the card inside the box is not yet 100% done either. Those should be small things to do though and once done, I can finish the layout of the parts and send them off to be cut.
-
Where are you putting the rig? If its in an outbuilding/shed have you considered making an exhaust manifold from the fans with large bore ducting to the outside world? That lot looks like it’ll kick out a serious amount if heat!
-
[quote name=“Kagemusha” post=“4395” timestamp=“1368963725”]
Where are you putting the rig? If its in an outbuilding/shed have you considered making an exhaust manifold from the fans with large bore ducting to the outside world? That lot looks like it’ll kick out a serious amount if heat!
[/quote]
Excellent question. I have a cellar, a large room in the basement that I share with a shop that stores some promotional materials there. It’s subterran and has a window that opens to the outside just below the ceiling. Even with the drier in the next room running on full blast, it’s always cold in that room. Right now I have half a dozen machines running there and it’s not gotten even remotely warmer in there.The rig will be setup so it blasts against the brick wall near the window (with a space between rig and wall, have to find the right distance once the rig is running). If heat get’s worse, I can replace the window with a lexan sheet and place fans in that to get the heat out of the room.
With one rig I don’t forsee any problems, the room is large and cool even in the hottest summer. I will keep an eye on temperatures though as I want the rig to run 24/7 for a very long time.
-
Nice. If you do run into problems and need some help with heat loss/gain I can run some quick calcs for you.
-
[quote name=“Kagemusha” post=“4398” timestamp=“1368965477”]
Nice. If you do run into problems and need some help with heat loss/gain I can run some quick calcs for you.
[/quote]
What kind of claculations? :) Any helps is always very wlecome. -
HVAC calcs. Basically taking into account building fabric, heat gain from equipment, heat loss through fabric of building, air flow, air temperature. Desired air temperature. Ill be able to calculate if you need/want any supplementary cooling/extraction.
I occasionally need to do calcs for rooms/buildings that are to house UPS and batteries/relays. Most other equipment I deal with isnt temperature sensitive.
-
[quote name=“Kagemusha” post=“4402” timestamp=“1368968359”]
HVAC calcs. Basically taking into account building fabric, heat gain from equipment, heat loss through fabric of building, air flow, air temperature. Desired air temperature. Ill be able to calculate if you need/want any supplementary cooling/extraction.I occasionally need to do calcs for rooms/buildings that are to house UPS and batteries/relays. Most other equipment I deal with isnt temperature sensitive.
[/quote]Sounds interesting. What numbers you need to do that? :)