My big mining rig project
-
I decided to do something insane and build a really big mining rig. I’ll document my journey on this as I go along.
The obligatory parts list first:
Mainboard: MSI Z77A-G45. It has 7 PCIE slots, 3 * 16x and 4 * 1x, which is one of the main reasons I got this one.
CPU: Intel Core i3 3220T 2.8Ghz, 2 Cores, 4 Threads. Simply something to make it run.
RAM: Crucial Ballistix Elite 2 x 4GB, DDR3-1866. Again, something to run it.
Power Supply: Club 3D CSP-X 1200CS, 1200 Watt. I wanted a different one but they ran out of stock and then they said that might not ever get back so I switched to this one.
Graphics cards: 7 times Club 3D HD 7950 `13Series 3GB.OS of choice is Ubuntu Server 13.04. I usually go Debian but I need some new packages that Debian just lags behind too much.
To start out I decided I would just try to get it running with one GPU to start with and then add GPU’s. The first setup was very openair because of that. I put Ubuntu on a USB stick to install it from and decided to go with a second stick as a disk replacement. Well it didn’t work out so good. If I tried booting the system with the USB stick in place for the system storage it wouldn’t budge. Removing the stick or putting it in a USB 2 instead of USB 3 slot would let the system boot. As soon as I plug it in USB 3, the system grinds to a halt until I remove it. On USB 2 the install started but it was slow as hell. After one hour of it trying to install packages (and it did work, just slow) I stopped the experiment. For now I scrapped that USB idea and used an old 1 TB SATA disk I had laying about.
Ubuntu installed in a breeze and then the troubles started. I couldn’t get cgminer to even configure. After much running in circles I finally made it compile but starting it would segfault. It took me a long while to figure it out, so here is the quick rundown, until I can write this up a bit better and more importantly, repeat the minimum steps needed to get there.
1. Install all the needed bits. There are things that are needed to build cgminer and some for ATI stuff to work, this is the command I have so far to make it work: “aptitude install x11-common xserver-xorg-core unzip libtool libncurses5-dev libjansson-dev libcurl4-openssl-dev build-essential git make gcc automake curl libudev-dev opencl-headers”
2. Downloading the cgminer sources. I used git to clone it to a folder.
3. Installing the catalyst drivers from AMD directly. I’m using the [url=http://www2.ati.com/drivers/linux/amd-catalyst-13.4-linux-x86.x86_64.zip]13.4 drivers 64bit[/url] version. Ã unzip it then run the installer. I had to run it once with “–force” option to get it to work, but I think that was because of my previous experiments. I chose all the standard options it threw at me and that did fine.
4. Download ADL_SDK and copying the files from include into the ADL_SDK folder of cgminer. The verison I got was ADL_SDK_5.0.zip
5. Download the APP_SDK. This is the file I have AMD-APP-SDK-v2.8-lnx64.tgz. There is another gz inside that one I had to unpack. Once I did I had a include/CL folder whose content I copied to /usr/include and the lib/x86_64 folder that I copied to /usr/lib. To finish it off I ran ldconfig.
6. Reboot for good measure.
7. Now I could run the cgminer configure like this: CFLAGS=“-O2 -Wall -march=native” ./configure --enable-scrypt
8. A make command later the executable was done and actually worked.I will run a couple tests and then continue with adding a second card.
-
Sounds fun but i have a couple of suggestions…
1. You’re gonna need 2 PSU’s jumpered together for 7 cards without a doubt else you’ll trip it.
2. You’ll struggle to keep it cool. I have 4 7970’s in an open air setup with fans/air-con blowing directly on them and still the 4th card is 10deg hotter than the 1st card. (cooling fans suck the hot air from the back of the other card and it accumulates.) Water-cooling is the solution but prohibitively expensive (and there’s no blocks for my cards). Interested to see how you overcome this.
3. You seriously need to consider how much power your PCI bus and your 24pin power connectors can handle - they WILL burnout without powered risers using 7 cards.
4. I have no experience with that board but i had an ASUS P6T6 WS Revolution with 6 PCIe and it wouldn’t ever recognize more than 2 cards. Someone else had the same issue too. Don’t be surprised if the same happens to you. It’s caused by lack of PCIe addresses or sommat.Good luck and keep us informed! 8)
-
Thanks for the feedback. I am ramping it up, and I have to wait either way since my riser cables are not here yet. Still dialing in one single card. Once that is done I will add card number 2. My biggest problem right now is ADL not working properly, so I can’t see what is happening from the inside. At this very moment I run it on Intensity 17 with 530 - 540 kh/s and a draw of the entire system of 300 - 315 Watts. So two power supplies might not even do it.
I’m monitoring the draw on the system very closely. I might burn the board once, but really try to avoid it. Using powered risers seems the way to go. I’ll get me a bunch soon.
The heat is an interesting point. Older cards were setup so they would disperse their heat through the back panel, but it seems that has changed significantly. With my hand I can feel that heat is pushed out through the fan, onto the board and then reflects out upwards (away from the laying mainboard). The heat I feel coming out the back is negligible. My original plan was to hang them from the bracket, but that would just reflect heat elsewhere. I have to go over my plan again and see if I can do something better.
On a side note, open systems are very bad for cooling. You just throw around air and waste most of your cooling potential. It’s the same as if you try to douse a fire with the garden hose on misting or on a stream. It’s the same pressure escaping but the result is much different (and yes I know it’s not a perfect analogy). Just as an experiment, use some cardboard or thin wood you have laying about, build a cone in front of the GPUs, even if its only a V shape towards them and keep everything else the same, then watch your temperatures. You’ll be surprised at how much even just small ducting will make a difference.
-
You will pull about 1000 watts with 4 cards running at 20 intensity.
[quote name=“ChristianRiesen” post=“2470” timestamp=“1368268621”]
Just as an experiment, use some cardboard or thin wood you have laying about, build a cone in front of the GPUs, even if its only a V shape towards them and keep everything else the same, then watch your temperatures. You’ll be surprised at how much even just small ducting will make a difference.
[/quote]Can you explain this in more detail please.
-
[quote name=“pyxis” post=“2474” timestamp=“1368269089”]
[quote author=ChristianRiesen link=topic=387.msg2470#msg2470 date=1368268621]
Just as an experiment, use some cardboard or thin wood you have laying about, build a cone in front of the GPUs, even if its only a V shape towards them and keep everything else the same, then watch your temperatures. You’ll be surprised at how much even just small ducting will make a difference.
[/quote]Can you explain this in more detail please.
[/quote]I too would like more details. i have spent many hours trying work out the best way to cool my rigs and the best effect i found so far was open air and 12" fan to blow cool air in to the cooling fans of the cards. Any other way just meant the cooling fans sucked in all the hot air from the back of the card next to it.
Heres a link to mine and Pyxis’ rigs. How would you suggest to improve cooling? http://forum.feathercoin.com/index.php?topic=264.50
-
It’s pulling 320 Watts with one card right now at intensity 20. Still fiddling with the settings. So far seems like intensity 20 is the way to go hands down. Ordering more power supplies now.
About the ducting, yes, that’s a general pet peeve whenever I see rigs. People just point normal room fans at them. That mitigates the problem lightly but not very effectively. Room fans are made for a broad effect, to move large masses of air, so it moves, which has a cooling effect all in itself. They are very bad at cooling a very specific point. Just run one and place your hand in front of it then feel where the stream drops off to the sides and then move a step back and do it again.
I’ll try to explain what I mean with a single GPU. Say you have a fan with a radius of 30 centimeters. Now in mid air in front of the fans direction hovers our GPU. Sure, somehow it is kept there, but let’s just assume it magically floats there for the sake of keeping it simple and to the important aspects. The tail is facing the fan, the front with the mounting bracket away from it. Most GPUs are designed to be cooled in that fashion, old ones even had a fan at the tail end that then blew over the entire card length. If the space around all of this is just empty, say nothing around it for a meter or so, then this will cool the card more than no fan, but not very efficiently. If the card is too close, the center of the fan creates a dead zone and it will even be less efficient than a bit away from the fan. Go too far and the effect is again almost nothing. You have to find the butterzone. Or you use a duct. Introduce a pipe, 30 cms diameter so the fan sits in it and the GPU behind it, still inside the pipe. Now the power of the fan is more evenly spread, even if the distance is a bit bigger. The moving air can’t diffuse and escape to the sides, there is pipe there as well. The increase in power (still the same fan, nothing changed there) is noticeable even if you had it in the butterzone before, in the open space. Now we take a huge vice and squeeze the pipe after the fan towards the card. The diameter is now half as much, the card just barely fits in there. The pressure form the fan is now bundled and cools the card much more efficiently. All that air around the card didn’t need to be moved around, it was wasted effort from the fan. Now all the fans power is directed towards cooling that card.
My example is a bit high level, but you can do simple things to reach similar goals. The main setup I see is a bunch of cards lined up next to one another. Now simply add some cardboard as a test (I would use something more solid, wood or metal) Just above and just below the are where the gpus are mounted to whatever frame you use. It will look like a V shape, with the opening away from the cards, getting narrower towards them. Make it large enough so your fan blows at them directly and isn’t able to blow over or below them. You could also tape the top and the bottom of the cardboard to the fan itself, which would increase the effect a bit. You could add sides to that ducting and through that have more air pressed into the area that counts.
If you fee very adventurous you could put a sheet below the cards so the air does get lined up more directly towards the mounting bracket. Above you could put one too, but keep two things in mind: Heat rises, so keep a bit of spacing and for a fire you only need air, fuel and heat, no spark. That cardboard is really bad for anything that gets too close to the GPUs.
Should you try it, please post some before and after numbers of the heat level :)
-
I understand completely and you are correct.
In my case though the issue comes from a space/fabrication/time/cost perspective. If i were to duct the 12" fan into a cone that is 6" there needs to be sufficient pressure to force the air through at a the higher velocity else you’ll just get a smaller radius of air. With high enough pressure of course you’ll double/triple your velocity and it would work a treat.
You’ll notice that the fans are placed 18" to 2ft away from the rigs as that seems to be the “butterzone” and avoids the dead spots in the center of the fan. Also 1 fan can cool 2 full rigs so noise levels are lower as are power costs. 8)
Just out of frame on that pic, i have a couple of 6" extractors which suck the hot air up blow it out of the shed (its all in my Log Cabin). They were originally rigged to duct cold air from outside to directly “through” the cards pretty much as you describe but even then the velocity wasn’t high enough to shift the hot air from one card away from the intakes on the next.
The other issue of course is the stoopid coolers on the windforce cards that just blow heat everywhere so it’s really difficult to move it away effectively and efficiently.
-
Yes, the on card fans are usually pretty dumb. I intend to build a metal duct that will mitigate the trouble there. Also will act as a heat sink in a fashion :)
Room fans are horrible for cooling. You would be better off getting better cooling fans that can get higher pressure, which would be in the same wattage area, though made to run 24/7. But even then, those fans are enough strong (usually, depends on what you have there) for small ducting. A 3:1 ratio is usually doable and you also have a “wide” spot to cool, not just a single spot. With that you will probably end up in a 2:1 ratio tops (fan area to back area of one rig).
-
Well, the room fans are currently doing a sterling job of keeping the cards at between 60 & 75 degrees so i’m happy with that.
look forward to seeing pics of your creation. :D
-
I keep my 3 x 6950’s outside in a metal shed, I have the warmest shed in Britain :D
It is often 20oC above ambient temps in there, but have not seen my cards break 70oC yet, the longterm heatsoak to mobo is astonishing though so I have a temp controlled extractor fan setup @ 30oC to the outside world, I use 2x PSU’s and am planning to tri-psu the rig and purchase 2 more 6950’s to help with the current hashrate. -
[quote name=“UKMark” post=“2489” timestamp=“1368275179”]
I keep my 3 x 6950’s outside in a metal shed, I have the warmest shed in Britain :D
[/quote]LOL - my brother is adamant that i’ll get raided by the cops coz the chopper will think i’m growing weed in there!
-
[quote name=“Nutnut” post=“2490” timestamp=“1368276831”]
[quote author=UKMark link=topic=387.msg2489#msg2489 date=1368275179]
I keep my 3 x 6950’s outside in a metal shed, I have the warmest shed in Britain :D
[/quote]LOL - my brother is adamant that i’ll get raided by the cops coz the chopper will think i’m growing weed in there!
[/quote]Let them raid me, I would love to see their faces when I have finished with my extrovertly geeky explanation :)
I am arresting you for (errrrrrr). You do not have to say anything, but it may harm your defence if you do not mention when questioned something which you later rely on in court. Anything you say may be given in evidence…
[i]…Satoshi made me do it. [/i] :D -
[quote name=“UKMark” post=“2496” timestamp=“1368278498”]
[quote author=Nutnut link=topic=387.msg2490#msg2490 date=1368276831]
[quote author=UKMark link=topic=387.msg2489#msg2489 date=1368275179]
I keep my 3 x 6950’s outside in a metal shed, I have the warmest shed in Britain :D
[/quote]LOL - my brother is adamant that i’ll get raided by the cops coz the chopper will think i’m growing weed in there!
[/quote]Let them raid me, I would love to see their faces when I have finished with my extrovertly geeky explanation :)
I am arresting you for (errrrrrr). You do not have to say anything, but it may harm your defence if you do not mention when questioned something which you later rely on in court. Anything you say may be given in evidence…
[i]…Satoshi made me do it. [/i] :D
[/quote]Haha, I think if you explained it into that sort of length they’d probably take you in on the grounds of ‘computer hacker’. Gotta love the UK police. =w=;
I’ll be running my 7970s in the outhouse (probably Tuesday onwards as my case/mobo haven’t been delivered today D: ) and when I ran just 1 6950 down there for BTC mining it became rather, cosy-warm. :P -
[quote name=“Nutnut” post=“2490” timestamp=“1368276831”]
LOL - my brother is adamant that i’ll get raided by the cops coz the chopper will think i’m growing weed in there!
[/quote]Was only thinking about this yesterday ;D
Some great ideas in these posts that I can use. My thanks to all.
-
Note to self: Use your own damn start scripts, you made them for a reason!
I was plagued with tons of HW errors. Turns out I simply didn’t run these lines after my reboots:
export DISPLAY=:0
export GPU_USE_SYNC_OBJECTS=1
export GPU_MAX_ALLOC_PERCENT=100*double facepalm*
-
So, some 3D Models here for your viewing pleasure.
[img]http://imageshack.us/a/img716/9792/miningrig01.jpg[/img]
A first general view. The blocks on the right are the power supplies, the one on the left the GPUS. The motherboard is behind the GPUs and those strange things at the bottom are two 230mm fans. There are these two metal sheets that angle towards the GPUs, that’s some ducting for airflow.
[img]http://imageshack.us/a/img560/5007/miningrig02.jpg[/img]
From the other side. The PSU on the left sucks in air through the bottom (large hole in the upper one) and pushes it out towards the back of the PSU (the lower one is in a standard position, the upper blows the hot air upwards). Also missing here is the back plate that holds all of this together. The mainboard is floating, as is the upper PSU. There is a plate that will cover the space from where the gpus are towards the back. I just removed it for displaying purposes.
[img]http://imageshack.us/a/img9/2857/miningrig03.jpg[/img]
Last but not least a close up of the ducting. The air gets pushed towards the cards. On each side (now open) there is another piece that stops air from going sideways, again removed to see everything easier. This setup also shields the mainboard from some extra heat. I was thinking about adding a side fan that just blows over the mainboard for a bit extra cooling of that area and to make sure not too much heat form the card gets to the mainboard.
Overall the rig is now 42 cm tall, 70 wide and 28 cm deep.
Not perfectly happy with it though. The cards are very close imho and that can lead to troubles. The cables will reach, they are at the most 16 cm from connector to plug, but it’s really stretching the outer limits of 19 cm that the cables can do. I still don’t have them in my hands, so that makes it a bit harder to fully plan this out.
I will store this version and start drawing up an alternative. Maybe I can come up with something even better.
-
Would placing a PSU on each side help?
Also to space the cards out more, you could use 2 pci riser cables for per card for cards furtherest away from the motherboard. -
Yes, placing one on each side was one of the modifications I had in mind. The cables seem to be rather short otherwise. Stacking risers is something I try to avoid, though I might not really need to, since we are talking about PCIE x1 here and the data that goes through there is rather low.
On another note: Powered risers seem to be a hand made thing, nobody sells them outright except cablesaurus and they don’t seem to do that anymore either. Looking at their picture, you can see very clear that it’s soldered on as an afterthought with only giving it some extra 12V juice. I have to say that sounds a bit odd to me, since GPUs draw the majority of their 12V power already from the external connectors. I maybe will run it on my other motherboard that has some nice tabs to read the voltage on the pcie bus easy. But that is for another time, once I have risers and maybe a powered one too. Either way, if you can’t buy one, it seems rather trivial to make one.
-
The journey continues.
To make everything a bit more stable I put the whole thing int an ATX case for now and added a second GPU. With two 7950’s running at 560 Kh/s each the system draws 580 - 590 Watts. Adding a third card would probably still be possible though pushing the PSU. Card 4 could mean disaster.
So I ordered some risers and cables to make powered risers out of them. I already have simple 1x risers on the way, which I could abuse (and destroy) for this task, since the x1 to x16 risers are just unconnected at the back part anyways.
A curious discovery on heat has happened though. Directly connected to the board, there is one free slot between these dual slot cards. Now the card whose fans are sucking in air from the other cards back seem to emit heat out back to the card below as well. Something I noticed in the single card setup, where there is heat leaking out around the fan, not just down and up. My design idea is now to create a heat barrier between cards.
-
Since I have the cards now in my hands and a better understanding and feel for them, I decided to redo the model for them in 3D. I also decided to leave off the bracket as it just hinders airflow with my new plans. Here a peek at how the model looks like:
[img]http://img198.imageshack.us/img198/3909/club3damdradeonhd7950no.jpg[/img]
Now for the next part, the heat shield.