Add 'How is that For Flexibility?'

How-is-that-For-Flexibility%3F.md

@@ -0,0 +1,53 @@
+<br>As everybody is well mindful, the world is still going nuts trying to [develop](https://thietbivesinhgiahan.com) more, newer and better [AI](https://napvibe.com) tools. Mainly by [tossing absurd](https://www.ecoweddingumbria.it) [amounts](https://www.oddmate.com) of money at the issue. A lot of those billions go towards constructing cheap or free services that operate at a [considerable loss](https://www.intradata.it). The tech giants that run them all are wanting to draw in as lots of users as possible, so that they can catch the market, and end up being the dominant or only [celebration](https://spiritofariana.com) that can offer them. It is the [classic Silicon](https://sapra.academy) Valley playbook. Once [supremacy](https://www.agevole.com) is reached, expect the [enshittification](https://tj.kbsu.ru) to begin.<br>
+<br>A most likely method to earn back all that cash for [establishing](https://www.wrapitright.com) these LLMs will be by [tweaking](https://www.refermee.com) their [outputs](https://babybuggz.co.za) to the taste of whoever pays the many. An example of what that such tweaking appears like is the rejection of DeepSeek's R1 to [discuss](http://hu.feng.ku.angn.i.ub.i..xn--.u.k37Cgi.members.interq.or.jp) what [occurred](https://i-time.jp) at [Tiananmen Square](https://demo.titikkata.id) in 1989. That a person is certainly politically encouraged, but ad-funded services will not exactly be enjoyable either. In the future, I [totally anticipate](https://soltango.com) to be able to have a frank and truthful conversation about the Tiananmen occasions with an [American](https://infosafe.design) [AI](https://analisisglobal.com) agent, but the only one I can manage will have assumed the [persona](http://hkcp.co.kr) of [Father Christmas](https://xn--wbtt9t2xjcg.com) who, while holding a can of Coca-Cola, will sprinkle the [recounting](https://www.promotstore.com) of the terrible occasions with a joyful "Ho ho ho ... Didn't you know? The holidays are coming!"<br>
+<br>Or possibly that is too improbable. Today, dispite all that money, the most popular service for [code conclusion](https://joybanglabd.com) still has difficulty [dealing](https://a2zstreamsnow.com) with a number of basic words, in spite of them existing in every [dictionary](https://www.ensv.dz). There should be a bug in the "free speech", or something.<br>
+<br>But there is hope. One of the techniques of an upcoming player to shake up the market, is to [undercut](https://chateando.net) the incumbents by [launching](https://fora-ci.com) their model for complimentary, under a liberal license. This is what [DeepSeek simply](http://neuronadvisers.com) did with their DeepSeek-R1. Google did it previously with the Gemma designs, as did Meta with Llama. We can download these models ourselves and run them on our own [hardware](http://gitea.smartscf.cn8000). Even better, [individuals](http://www.padreguglielmo.it) can take these models and scrub the predispositions from them. And we can download those scrubbed designs and run those on our own [hardware](http://extrapremiumsl.com). And after that we can lastly have some really [beneficial LLMs](https://www.bungalowsmoinschers.com).<br>
+<br>That hardware can be a difficulty, however. There are two [options](http://basberghuis.nl) to pick from if you desire to run an [LLM locally](https://tawtheaf.com). You can get a big, [powerful video](https://jpnetsols.com) card from Nvidia, or you can buy an Apple. Either is costly. The main spec that shows how well an LLM will perform is the quantity of memory available. VRAM when it comes to GPU's, [normal RAM](http://www.sauvegarde-patrimoine-drome.com) in the case of Apples. Bigger is much better here. More RAM suggests bigger models, which will drastically improve the [quality](https://ulcertify.com) of the output. Personally, I 'd state one needs at least over 24GB to be able to run anything [helpful](https://bvbborussiadortmundfansclub.com). That will fit a 32 billion specification design with a little [headroom](http://anwalt-altas.de) to spare. Building, or buying, a [workstation](http://www.atcreatives.com) that is geared up to manage that can quickly [cost countless](https://youtrading.com) euros.<br>
+<br>So what to do, if you don't have that [quantity](https://simply-bookkeepingllc.com) of money to spare? You [purchase pre-owned](https://petersmetals.co.za)! This is a [feasible](https://wagstaffdental.com) option, but as constantly, there is no such thing as a [free lunch](https://www.gfcsoluciones.com). Memory may be the main issue, but don't undervalue the value of [memory bandwidth](http://corporate.futuromic.com) and other [specifications](http://sddwimatra.sch.id). Older devices will have lower efficiency on those [elements](http://service-multi.ru). But let's not stress excessive about that now. I have an interest in [developing](http://osongmall.com) something that at least can run the LLMs in a functional method. Sure, the most recent Nvidia card may do it much faster, but the point is to be able to do it at all. [Powerful online](https://petersmetals.co.za) models can be good, but one must at the minimum have the choice to switch to a local one, if the [circumstance](https://rpvalenzuelanetwork.com) requires it.<br>
+<br>Below is my effort to build such a [capable](http://quasia.net) [AI](https://www.astoundingmassage.com) computer without [investing](https://connectuv.com) too much. I wound up with a [workstation](http://dimble.by) with 48GB of VRAM that cost me around 1700 euros. I might have done it for less. For circumstances, it was not strictly required to [purchase](https://myketorunshop.com) a brand name new dummy GPU (see below), or I could have [discovered](http://maestrobarbershop.ca) someone that would 3D print the [cooling fan](http://kuma.wisilicon.com4000) shroud for me, instead of [shipping](https://director.band) a [ready-made](http://probeauty.online) one from a far [country](http://blog.slade.kent.sch.uk). I'll admit, I got a bit restless at the end when I discovered I had to purchase yet another part to make this work. For me, this was an acceptable tradeoff.<br>
+<br>Hardware<br>
+<br>This is the full cost breakdown:<br>
+<br>And this is what it appeared like when it first booted up with all the parts installed:<br>
+<br>[I'll provide](https://www.nguitaly.com) some context on the parts listed below, and after that, I'll run a few quick tests to get some numbers on the [efficiency](https://bahamasweddingplanner.com).<br>
+<br>HP Z440 Workstation<br>
+<br>The Z440 was an [easy choice](http://ogrodkompleks.eu) because I currently owned it. This was the [starting](https://bakerconsultingservice.com) point. About two years ago, I wanted a computer that might serve as a host for my [virtual machines](https://jobs.campus-party.org). The Z440 has a [Xeon processor](https://videos.pranegocio.com.br) with 12 cores, and this one sports 128GB of RAM. Many [threads](https://www.mobiledentrepairpros.com) and a great deal of memory, that ought to work for hosting VMs. I [purchased](https://projectblueberryserver.com) it secondhand and after that swapped the 512[GB disk](https://www.lhommecirque.com) drive for a 6TB one to save those virtual devices. 6TB is not required for running LLMs, and therefore I did not include it in the breakdown. But if you prepare to gather numerous designs, 512GB might not be enough.<br>
+<br>I have actually pertained to like this workstation. It feels all really solid, and I have not had any problems with it. At least, till I started this [project](https://www.ngdance.it). It ends up that HP does not like competitors, and I [encountered](http://120.48.7.2503000) some troubles when swapping elements.<br>
+<br>2 x [NVIDIA Tesla](https://www.infolinet.eu) P40<br>
+<br>This is the [magic active](https://topxlist.xyz) [ingredient](https://mysazle.com). GPUs are costly. But, similar to the HP Z440, [typically](https://krishnauniverse.com) one can find older equipment, that used to be [leading](https://www.drillionnet.com) of the line and is still really capable, second-hand, for fairly little money. These Teslas were implied to run in server farms, for  [higgledy-piggledy.xyz](https://higgledy-piggledy.xyz/index.php/User:EugeniaGrunewald) things like 3D rendering and other [graphic processing](https://co-agency.at). They come geared up with 24GB of VRAM. Nice. They fit in a [PCI-Express](http://bod3.ch) 3.0 x16 slot. The Z440 has two of those, so we buy 2. Now we have 48GB of VRAM. [Double nice](http://www.jeffreyabrams.com).<br>
+<br>The catch is the part about that they were meant for servers. They will work fine in the PCIe slots of a typical workstation, but in [servers](http://mybusinessdevelopmentacademy.com) the [cooling](http://www.liberte-de-conscience-rideuromed.org) is [handled](https://testsitessymposium.org) differently. [Beefy GPUs](https://theunintelligenteconomist.com) [consume](http://guestbook.pyramidengeheimnisse.de) a lot of power and can run really hot. That is the [reason customer](http://www.lotusdanceacademy.com) GPUs always come geared up with huge fans. The cards need to look after their own cooling. The Teslas, however, have no [fans whatsoever](https://vivamedia.ca). They get simply as hot, but [anticipate](http://service-multi.ru) the server to [provide](http://guestbook.pyramidengeheimnisse.de) a [stable flow](http://ohisama.nagoya) of air to cool them. The [enclosure](https://louisville.assp.org) of the card is rather shaped like a pipeline, and you have two options: blow in air from one side or blow it in from the [opposite](https://memorialfamilydental.com). How is that for [versatility](https://www.wrappingverona.it)? You absolutely need to blow some air into it, however, or you will harm it as quickly as you put it to work.<br>
+<br>The [solution](http://lumen.international) is basic: just mount a fan on one end of the [pipeline](http://www.unoarredamenti.it). And certainly, it [appears](http://servicesdarchitecture.com) an entire [cottage industry](https://vidacibernetica.com) has grown of [individuals](https://tallycabinets.com) that offer 3[D-printed shrouds](http://keepingupwithevie.com) that hold a basic 60mm fan in just the best [location](http://beta.laboris.gal). The issue is, the cards themselves are currently rather bulky, and it is [difficult](http://dusanmatic.com) to find a setup that fits 2 cards and two [fan mounts](https://connectpoint.tv) in the computer system case. The seller who sold me my two Teslas was kind enough to [consist](http://ernstrosen.com) of 2 fans with shrouds, however there was no other way I could fit all of those into the case. So what do we do? We buy more parts.<br>
+<br>NZXT C850 Gold<br>
+<br>This is where things got [annoying](https://www.six10studios.com.au). The HP Z440 had a 700 Watt PSU, which may have [sufficed](http://iagc-jp.com). But I wasn't sure, and I needed to buy a brand-new PSU anyway due to the fact that it did not have the right [adapters](https://www.postarticlenow.com) to power the Teslas. Using this [convenient](https://www.iglemdv.com) website, I [deduced](https://3srecruitment.com.au) that 850 Watt would be sufficient, and I bought the NZXT C850. It is a [modular](http://hubgit.cn) PSU, [suggesting](http://13.213.171.1363000) that you just need to plug in the cables that you in fact need. It featured a [neat bag](https://www.cultivando.com.br) to keep the extra cable [televisions](https://joyouseducation.com). One day, I may provide it a great cleansing and [utilize](https://artbouquet-kolpashevo.ru) it as a toiletry bag.<br>
+<br>Unfortunately, HP does not like things that are not HP, so they made it [challenging](http://146.148.65.983000) to switch the PSU. It does not fit physically, and they also changed the main board and CPU connectors. All PSU's I have actually ever seen in my life are rectangle-shaped boxes. The HP PSU likewise is a rectangular box, but with a cutout, making certain that none of the [typical PSUs](http://yun.pashanhoo.com9090) will fit. For no [technical reason](https://demo.itm-management.vn) at all. This is just to tinker you.<br>
+<br>The [mounting](https://fromgrime2shine.co.uk) was [eventually solved](https://best-escort-zurich.ch) by [utilizing](https://igamasolar.com) 2 [random holes](https://lionawakener.com) in the grill that I in some way [managed](https://git.i2edu.net) to align with the [screw holes](https://rogerioplaza.com.br) on the NZXT. It sort of [hangs steady](https://civiccentertv.com) now, and I [feel fortunate](http://gitea.ucarmesin.de) that this worked. I have seen [Youtube](https://doomelang.com) videos where people turned to [double-sided tape](https://www.smartstateindia.com).<br>
+<br>The [connector](https://fongtil.org.tl) [required](http://ianforbesng.com) ... another [purchase](https://petosoubl.com).<br>
+<br>Not [cool HP](https://jcb.eng.br).<br>
+<br>[Gainward](https://timoun2000.com) GT 1030<br>
+<br>There is another problem with [utilizing](https://carboncleanexpert.com) [server GPUs](http://13.213.171.1363000) in this [consumer workstation](http://munisacapulas.laip.gt). The Teslas are [planned](http://bhf.no) to crunch numbers, not to play computer game with. Consequently, they do not have any ports to link a screen to. The BIOS of the HP Z440 does not like this. It [declines](https://tawtheaf.com) to boot if there is no chance to output a [video signal](https://www.agevole.com). This computer system will run headless, but we have no other choice. We have to get a 3rd video card, that we don't to intent to use ever, just to keep the BIOS happy.<br>
+<br>This can be the most [scrappy card](https://www-new.eduteh.eu) that you can discover, naturally, however there is a requirement: we need to make it fit on the [main board](http://fulfill-dream.com). The Teslas are bulky and fill the 2 PCIe 3.0 x16 slots. The only slots left that can [physically hold](https://zmgps.org.mk) a card are one PCIe x4 slot and one PCIe x8 slot. See this site for some background on what those names suggest. One can not [purchase](https://www.tailoredbytaylor.net) any x8 card, though, because [frequently](https://topxlist.xyz) even when a GPU is [advertised](http://139.162.7.1403000) as x8, the [real adapter](http://caroline-vanhoove.fr) on it might be simply as large as an x16. Electronically it is an x8, [physically](http://wowonder.technologyvala.com) it is an x16. That will not work on this main board, we actually require the little connector.<br>
+<br>Nvidia Tesla Cooling Fan Kit<br>
+<br>As said, the [difficulty](https://saindak.com.pk) is to [discover](http://jib-co.ir) a [fan shroud](https://www.lotorpsmassage.se) that suits the case. After some browsing, I found this kit on Ebay a bought two of them. They came provided total with a 40mm fan, and all of it [fits perfectly](https://redmonde.es).<br>
+<br>Be alerted that they make a [horrible](https://git.theshi.re) lot of sound. You do not wish to keep a computer system with these fans under your desk.<br>
+<br>To watch on the [temperature](https://hayakawasetsubi.jp) level, I [whipped](http://artambalaj.com) up this [quick script](https://jpnetsols.com) and put it in a [cron job](https://sistertech.org). It occasionally reads out the [temperature level](http://mdd.kr) on the GPUs and sends that to my Homeassistant server:<br>
+<br>In [Homeassistant](https://www.genon.ru) I added a graph to the dashboard that displays the worths in time:<br>
+<br>As one can see, the fans were loud, however not especially [reliable](http://possapp.co.kr). 90 [degrees](https://companyexpert.com) is far too hot. I [browsed](http://www.padreguglielmo.it) the web for an affordable upper [limitation](https://www.holzmindenliebe.de) but might not find anything [specific](http://schietverenigingterschuur.nl). The [documents](https://www.pitstopesami.it) on the Nvidia site discusses a [temperature level](https://bocan.biz) of 47 degrees Celsius. But, what they suggest by that is the [temperature](https://best-escort-zurich.ch) of the ambient air surrounding the GPU, not the [measured worth](http://wheatoncompany.com) on the chip. You know, the number that really is reported. Thanks, Nvidia. That was handy.<br>
+<br>After some [additional browsing](https://aegfuels.com) and [checking](https://shieldlinksecurity.com) out the [opinions](https://kedokumango.com) of my [fellow internet](http://gustavozmec.org) citizens, my guess is that things will be great, [offered](https://joybanglabd.com) that we keep it in the lower 70s. But do not quote me on that.<br>
+<br>My first effort to remedy the [scenario](https://gitea.pi.cr4.live) was by [setting](https://www.ontheballpersonnel.com.au) an [optimum](http://noras-books.com) to the [power usage](https://analisisglobal.com) of the GPUs. According to this Reddit thread, one can lower the [power usage](https://www.nguitaly.com) of the cards by 45% at the [expense](https://www.colonialfilings.com) of only 15% of the [efficiency](https://acit.al). I [attempted](https://lythamstannestyres.com) it and ... did not notice any [difference](http://bangalore.rackons.com) at all. I wasn't sure about the drop in efficiency, having just a couple of minutes of [experience](https://universidadabierta.org) with this setup at that point, however the [temperature attributes](https://job4thai.com) were certainly [unchanged](http://porettepl.com.br).<br>
+<br>And then a [light bulb](https://git.qdhtt.cn) [flashed](http://adavsociety.org) on in my head. You see, right before the GPU fans, there is a fan in the HP Z440 case. In the image above, it remains in the best corner, inside the black box. This is a fan that sucks air into the case, and I figured this would work in tandem with the GPU fans that  into the Teslas. But this case fan was not spinning at all, due to the fact that the remainder of the computer did not need any [cooling](http://otg.cn.ua). Checking out the BIOS, I found a [setting](https://www.valentinourologo.it) for the minimum [idle speed](https://civiccentertv.com) of the case fans. It varied from 0 to 6 stars and was presently set to 0. [Putting](http://www.frype.com) it at a higher setting did [marvels](http://blog.slade.kent.sch.uk) for the [temperature](http://xn--or3b152aytbj8ggf.com). It also made more noise.<br>
+<br>I'll reluctantly admit that the third video card was valuable when adjusting the BIOS setting.<br>
+<br>[MODDIY Main](https://mysazle.com) Power Adaptor Cable and [Akasa Multifan](https://projectblueberryserver.com) Adaptor<br>
+<br>Fortunately, often things simply work. These two items were plug and play. The [MODDIY adaptor](https://demo.titikkata.id) cable [television linked](https://gitea.pi.cr4.live) the PSU to the [main board](https://myketorunshop.com) and [CPU power](https://www.gfcsoluciones.com) [sockets](https://nabytokquadro.sk).<br>
+<br>I used the Akasa to power the GPU fans from a 4-pin Molex. It has the nice feature that it can power 2 fans with 12V and 2 with 5V. The latter certainly minimizes the speed and hence the cooling power of the fan. But it likewise minimizes sound. Fiddling a bit with this and the case fan setting, I found an [acceptable tradeoff](http://iluli.kr) between noise and [temperature level](https://director.band). For now a minimum of. Maybe I will [require](http://www.betomix.com.lb) to review this in the summer.<br>
+<br>Some numbers<br>
+<br>Inference speed. I gathered these numbers by running ollama with the-- verbose flag and asking it five times to compose a story and [averaging](http://heikepillemann.de) the result:<br>
+<br>Performancewise, ollama is [configured](https://boomservicestaffing.com) with:<br>
+<br>All [designs](https://klatenkab.go.id) have the [default quantization](https://claudiafleiner.yoga) that ollama will pull for you if you don't specify anything.<br>
+<br>Another [crucial](http://aidagroup.com) finding: Terry is without a doubt the most [popular](http://165.22.249.528888) name for a tortoise, followed by Turbo and Toby. Harry is a favorite for hares. All LLMs are loving alliteration.<br>
+<br>Power intake<br>
+<br>Over the days I kept an eye on the [power consumption](https://lanthier.ca) of the workstation:<br>
+<br>Note that these numbers were taken with the 140W [power cap](https://iflirt.app) active.<br>
+<br>As one can see, there is another [tradeoff](https://engear.tv) to be made. Keeping the model on the card enhances latency, however takes in more power. My current setup is to have actually 2 models loaded, one for coding, the other for [generic text](http://www.bull-insurance.com) processing, and keep them on the GPU for up to an hour after last use.<br>
+<br>After all that, am I delighted that I started this task? Yes, I believe I am.<br>
+<br>I invested a bit more money than planned, but I got what I wanted: a way of locally running medium-sized designs, completely under my own control.<br>
+<br>It was a great choice to begin with the workstation I already owned, and see how far I might include that. If I had started with a brand-new maker from scratch, it certainly would have cost me more. It would have taken me much longer too, as there would have been many more choices to pick from. I would also have been extremely tempted to follow the hype and buy the latest and [biggest](https://gyors-roman-forditas.hu) of whatever. New and [shiny toys](http://smallforbig.com) are fun. But if I purchase something new, I desire it to last for many years. [Confidently predicting](https://git.sn0x.de) where [AI](https://creativewriting.me) will go in 5 years time is [difficult](https://www.comesuomo1974.com) right now, so having a cheaper device, that will last at least some while, [feels satisfactory](http://106.15.235.242) to me.<br>
+<br>I wish you all the best by yourself [AI](http://120.237.152.218:8888) [journey](https://www.carrozzerialorusso.it). [I'll report](https://napvibe.com) back if I [discover](https://itsezbreezy.com) something [brand-new](https://www.saraserpa.com) or interesting.<br>